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Executive Summary 


The Literacy Design Collaborative (LDC) was created to support teachers in implementing 
Common Core State Standards (CCSS) and embedding literacy skill development throughout 
content area curriculum. The LDC Investing in Innovation (i3) project focuses on developing 
teacher competencies through job-embedded professional development and the use of 
professional learning communities (PLCs). Teachers work collaboratively with coaches to 
develop their expertise further and design standards-driven, literacy-rich writing assignments 
within their existing curricula across all content areas. 


Engaged in the evaluation of LDC tools since June 2011, UCLA’s National Center for 
Research on Evaluation, Standards, and Student Testing (CRESST) is the independent evaluator 
for LDC’s federally funded i3 validation grant. CRESST’s evaluation is using multiple data sources 
and a quasi-experimental design (QED) to examine LDC implementation and impact in two 
cohorts of schools in a large, urban West Coast school district. Study schools serve largely 
Hispanic populations, with a high proportion of students qualifying for free and reduced lunch, 
and many English language learners. 


This annual report reflects the second year of implementation in 14 Cohort 1 schools, 
which began implementation during 2016-2017, and the first year of implementation for 31 
Cohort 2 schools, which commenced at the beginning of the 2017-2018 school year. The 
program experienced a high level of attrition among Cohort 1 participants and less than a 
quarter of the total 2017-2018 participants are teachers and administrators from Cohort 1. 
With so many new participants and program effects on student learning expected only after 
two years of implementation, this annual report should be considered an interim look at LDC 
progress. It is also noteworthy that nearly two thirds of all 2017-2018 participating teachers 
taught at the elementary school level, as that development represents an expansion of LDC’s 
original focus on middle schools. 


The CRESST evaluation addresses research questions in three major areas: 
e Program Characteristics and Implementation 


e Contextual Factors and Implementation 


e Program Impacts 


The following findings draw on multiple methods. These include surveys of teachers, 
teacher leaders, and administrators; the analysis of LDC modules, which are a central 
manifestation of LDC practice; participant interactions with LDC CoreTools, the electronic 
platform through which teachers access LDC professional development resources (online 
courses, existing LDC modules, module templates, and support for module development); LDC 
administrative records; and administrative data on students and teachers including class 
rosters, student demographics, and student performance on state standards-based 
assessments. We begin with overall findings and then move to summarize perspectives on key 
LDC components, intermediate effects on teachers’ instructional strategies and practice, and 
effects on student outcomes. Detailed evidence with regard to key LDC activities, supports, and 
pedagogical impacts help to explain these positive findings and offer implications for further 
strengthening LDC. 


Overall Findings 


Findings from both participant surveys and analyses of student outcomes reveal positive 
results for the LDC intervention: 


e Teacher, teacher leader, and administrator participants reported positive attitudes 
toward LDC. All measures of satisfaction or improvement were rated positively by 
more than half of the respondents, and ratings appeared higher this school year than 
last. More than half of the teachers expressed interest in learning more about how to 
lead LDC implementation at their schools, and 70% of teacher leaders and all 
administrators anticipated that their teachers would continue with LDC the following 
year. 


e Participants across all groups perceived a positive impact on student outcomes. 
Approximately three quarters of both teachers and administrators agreed that LDC 
helped improve students’ college and career readiness, literacy performance, writing, 
and content knowledge. The two most highly rated areas of impact, according to all 
three groups, were students’ ability to complete writing assignments and quality of 
students’ writing. 


e Analysis of student outcomes confirmed participants’ positive views. QED 
comparisons of treated and matched students revealed that LDC had statistically 
significant effects on Cohort 2 middle school students’ English language arts (ELA) 
performance, with an effect size of .15 for students receiving LDC instruction in all 
three core content areas: ELA, science, and social studies/history. No statistically 
significant differences were found for students taught by Cohort 2 elementary school 
teachers or Cohort 1 returning middle school teachers. 


Professional Learning Community and Teacher Collaboration 


e Nearly all LDC teachers participated in LDC-oriented PLCs and approximately three 
quarters reported that their PLCs met at least every other week, the frequency 
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expected by the LDC program. For teachers, the most cited barrier to meeting every 
week was that PLC members had other priorities that competed with their 
participation. For teacher leaders the most cited barrier was that PLC time was not 
protected. 


e The frequency with which PLCs met varied across schools and teachers’ individual 
attendance rates also varied greatly. Although a majority of schools met attendance 
goals, a substantial number of schools experienced challenges related to protecting 
common planning time and ensuring that teacher participants regularly attended PLC 
meetings. 


e Both teacher surveys and PLC reflection forms indicated that PLC meetings typically 
lasted 45 minutes to an hour or an hour or more, again generally meeting the LDC 
standard. 


e Teachers valued the collaborative nature of LDC and its PLCs. A large majority of 
teachers credited LDC with making them more likely to collaborate with other 
teachers, not only within their grade levels and content areas but outside of them as 
well. 


e Almost two thirds of teachers reported that they had sufficient time to meet during 
PLC sessions, and about three quarters felt that their administrators protected 
allocated resources to ensure that they could meet. However, the most frequently 
requested modification for future years was for more dedicated time during and 
outside of PLCs to plan modules, implement, and receive feedback about 
implementation. Teachers also requested that LDC time be compensated, and that 
their schools should invest in substitute coverage to allow for more professional 
development time. 


LDC Training and Support 


e Teachers were nearly uniform in their positive attitudes about the value of their PLC 
participation. They found the PLCs a safe space for sharing instructional plans, 
problem solving, and learning to develop modules. 


e Teacher leaders were almost universally reported to be highly approachable, 
supportive, knowledgeable, and helpful, which suggests that programmatic changes 
made for 2017-2018 were successful. During the 2017-2018 school year, local PLC 
leadership was assigned to the teacher leaders rather than the project liaisons, the 
teacher leaders were provided with a stipend, and principals and assistant principals 
were no longer permitted to play the role. 


e Teacher leaders reported high satisfaction regarding the support they received from 
coaches, professional development offerings, and how the teacher leader role allowed 
them to be instructional leaders in their schools. 


e Overall, LDC coaches received positive feedback on the survey, with 93% of teachers 
and 98% of teacher leaders reporting that their coaches gave them appropriate and 
timely feedback and support. Data, however, suggest room for improvement when it 
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comes to frequency and usefulness of coach feedback. Analysis of CoreTools analytic 
data revealed that many modules did not receive comments from coaches, and peer 
review functionality was used infrequently. Nearly half of teachers reported not using 
written feedback from their coaches, although those that did tended to find it useful. 


Most teachers rated CoreTools positively, demonstrating the success of changes LDC 
made prior to the 2017-2018 school year including further refinement of the content, 
sequencing, delivery of CoreTools’ instructional content, and streamlining of 
participants’ learning process. A majority of teachers found the online course 
materials clear, relevant, and useful and, contrary to last year, open-ended responses 
did not reference instructional content as a problem. Almost half of teachers, 
however, rated the ease of use of the online course material as either poor or fair, and 
a similar proportion did not find the videos useful. In addition, analysis of CoreTools 
data revealed that teachers were being exposed to a small proportion of the online 
course content, which suggests that there wasn’t sufficient PLC time to cover the 
content LDC intended participants to engage with. 


While 87% of teachers agreed that their school had adequate technology to support 
LDC implementation, many teachers noted in their open-ended responses that there 
were glitches in conducting Zoom meetings. Teachers also suggested the value of 
more in-person time with coaches. 


LDC Implementation 


Teachers reported adapting and/or creating at least two LDC modules on average in 
addition to selecting and using mini-tasks outside of the modules, meeting LDC 
program expectations. According to the survey data, more than 90% of teachers 
reported implementing at least two modules in their classrooms. Analysis of CoreTools 
data, however, show that only about one third of teachers uploaded student work for 
two or more modules (an important proxy for classroom implementation). 


Analysis of program data suggests that while nearly all participants were engaging 
with the module-building platform, the level of engagement did vary greatly across 
individuals and across subgroups (role, cohort, school level, content area) as 
evidenced by the large variation in the number of views, edits, and comments. More 
in-depth analysis of the portions of the modules teachers were editing confirmed the 
finding that engagement varied greatly across teacher participants, with about half of 
teachers failing to engage at a basic level by editing the teaching task, and other 
teachers engaging more deeply by editing multiple portions of modules. 


The majority of teachers (71% to 92%) reported success in nine key areas of LDC 
module development. Teachers were most confident in selecting focus standards, 
creating the writing assignment, identifying skills needed in the module, and making 
writing assignments relevant and engaging. The module analysis, however, suggests 
that the materials adapted and created by PLC members varied in levels of completion 
and quality and using uploaded student work as an indicator, it appears that a sizable 
proportion of modules might not have been implemented in the classroom. 
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With regard to their classroom implementation of LDC modules, the majority of 
teachers reported success with all six key areas queried (79% to 87%). Teachers were 
most confident with engaging students in complex text, locating evidence of standards 
in final student work, and engaging students in understanding the assignment and 
rubric. 


Ratings of module quality increased from last school year to this school year, with 
elementary modules appearing relatively stronger than the secondary school 
modules. 


Leadership Support 


Almost all teachers and teacher leaders reported that their administrators encouraged 
LDC participation at the school. The majority of teachers and teacher leaders agreed 
with administrators that they allocated resources to ensure that LDC teachers could 
participate in meetings. Administrators generally voiced strong support for LDC, but 
there were differences with how active teachers and teacher leaders viewed that 
support. LDC may want to understand the differences in these perceptions better. 


Overall, both administrators and teacher leaders took advantage of in-person 
meetings offered by LDC. There was great variation, however, in how frequently 
teacher leaders met over the phone with coaches for planning purposes. 


Compared to uniformly positive ratings by school administrators on most items, their 
ratings of district support were relatively low. This too signals an area where LDC may 
need to strengthen its efforts. 


Impact on Teacher Practice 


The majority of teachers reported improving their practice in seven LDC-related skills 
(75% to 84%). Teachers felt they learned most in selecting focus standards, creating 
standards-driven writing assignments, and identifying skills students need in writing 
assignments (skills concentrated at the beginning of the LDC learning cycle). 


Approximately 80% of teachers agreed that participating in LDC raised their 
expectations for students’ writing, helped them incorporate writing assignments into 
their existing curriculum, and made them more likely to collaborate with other 
teachers on designing instruction. 


Subgroup Effects 


Cohort 1 new teachers’ survey responses were less enthusiastic than their peers on a 


number of dimensions of LDC training, support, implementation and impact. These dimensions 
included the effectiveness of the PLC, usefulness of coach feedback, and teachers’ own capacity 


to be a part of shaping the initiative. CoreTools analyses also showed that Cohort 1 new 


teachers demonstrated less engagement in terms of viewing, editing, and commenting on 
modules. These findings may signal the challenge of new teachers entering existing PLCs. These 


teachers may have felt less sense of belonging and/or that they were trailing behind teachers 


who had participated the prior year. Oddly however, Cohort 1 new teachers’ modules were 


rated more highly than other teachers and Cohort 1 returning teachers rated relatively the 
lowest, although sample size and representativeness limit any generalization. 


There were few differences by school level. Despite this, Cohort 2 elementary school 
teachers appeared more confident in their ratings and the elementary school modules 
appeared stronger than did those developed by teachers at the secondary school level. A rare 
instance of secondary school advantage was in the area of administrator support and 
involvement. 


The fidelity of implementation analysis did not reveal differences in average overall levels 
of fidelity between Cohort 1 and 2 schools, and between elementary and middle schools. 


Conclusions 


LDC’s second year of implementation shows the results of its continuous improvement 
efforts and a positive trajectory toward program success. Changes made based on last year’s 
evaluation results—such as the further streamlining and refining of online courses and the 
changing of the school liaison role to that of a teacher leader—have paid off in greater 
participant satisfaction and greater fidelity to the program’s logic model. 


Evidence suggests that second year implementation was largely faithful to the structure of 
LDC’s implementation model, although in many cases individual teachers and schools were not 
meeting implementation goals. Most PLCs met at least every other week as they were expected 
to do, but a sizable minority struggled to protect common planning time and ensure high 
attendance rates. Participants engaged with CoreTools resources, but to varying degrees. 
Coaches provided feedback through multiple mechanisms, and PLC members reported 
appreciating this feedback, but feedback was not uniform and not all teachers found it useful. 
Teachers also reported meeting LDC expectations for module development and 
implementation, although data from the CoreTools analyses suggest that, for some teachers, 
module development and implementation may not have been fully complete. 


The data from all respondents’ surveys, coupled with the analysis of module quality, 
suggest that LDC is benefiting most teachers’ pedagogical skills and particularly their ability to 
integrate standards-based writing assignments and assessment into their curriculum. Outcome 
results also are promising. Not only do all respondents perceive LDC’s positive results on 
student learning—particularly with regard to literacy skills and writing—but performance on 
the state ELA test revealed a statistically significant, positive effect for Cohort 2 middle school 
students. 


This year’s results revealed some areas of challenge. While the majority of participants 
were Satisfied and positive about the benefits of LDC, significant proportions were not ina 
number of areas. For example, while the great majority of teachers were confident and noted 
improvement in areas related to creating standards-based teaching and writing tasks, and 
evaluating them, sizable minorities were less confident about the specifics of planning lessons 


and practicing formative assessment. LDC should consider what can be done do to strengthen 
their efforts in these areas and/or target further follow-up support for teachers during the 
coming year. 


Adequate time was a frequently cited concern. Respondents voiced that they hoped for 
more collaborative time, more prep time, time before the start of the year, and/or 
compensated time. Technology also remained a problem for some participants. While some 
respondents were frustrated with technical problems that marred the experience of Zoom 
meetings, others would have simply preferred in-person rather than digital meetings with 
coaches. Participants also seem to have made relatively limited use of the CoreTools 
interactivity, for example with regard to commenting and revision. Some participants also 
reported frustration with CoreTools usability and the utility of its video resources. 


Survey responses also suggest that leadership support needs continuing attention. 
Although teachers and teacher leaders were generally positive about their school leaders’ 
knowledge of and support for LDC, these leaders did not seem to meet LDC’s expectations for 
being involved in program implementation by observing and providing feedback to participants. 
Similarly, administrator responses suggested that district support was a relative weakness. 


Finally, one new issue that surfaced this school year was the challenge faced by new 
teacher participants joining established PLCs. Survey responses and CoreTools analyses showed 
that teachers who recently joined PLCs at Cohort 1 schools in 2017—2018 appeared less 
satisfied and less engaged in LDC than did their peers. LDC may want to inquire further into the 
reason for these findings and/or consider how to facilitate PLC entry for new participants, and 
how to differentiate program resources and support to meet their needs. 


As we noted, this report presents interim findings, before most participants have had the 
two years of experience with the program that LDC believes they need to meet program goals. 
The findings at this point are definitely positive. The 2018-2019 school year will provide that 
crucial second year of experience and, barring unexpected attrition, strong data for fully 
examining the implementation and effects of LDC’s i3 implementation, and the factors that 
contribute to LDC success. 
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1.0 Introduction 


The Literacy Design Collaborative (LDC) was created to support teachers in implementing 
Common Core State Standards (CCSS) by integrating literacy skill development throughout the 
content areas. The LDC Investing in Innovation (i3) project focuses on developing teacher 
competencies through job-embedded professional development and the use of professional 
learning communities (PLCs). Teachers work collaboratively with coaches to develop their 
expertise further and design standards-driven, literacy-rich writing assignments within their 
existing curriculum across all content areas. LDC is a national community of educators providing 
a teacher-designed and research-based framework, online tools, and resources for creating 
both literacy-rich assignments and courses across content areas. Used by individual teachers, 
schools, and districts in 40 states for the past four years, LDC also is a statewide-adopted 
strategy for Common Core implementation in Kentucky, Colorado, Louisiana, and Georgia. 


UCLA’s National Center for Research on Evaluation, Standards, and Student Testing 
(CRESST), in collaboration with its partner Research for Action (RFA), engaged in the evaluation 
of LDC implementation and its impact on student learning and teacher effectiveness starting in 
June 2011, via two parallel research studies funded by the Bill and Melinda Gates Foundation. 
Those studies included an examination of LDC implementation in eighth grade social studies 
and science classrooms in Kentucky and Pennsylvania and a districtwide implementation in 
sixth grade advanced reading classes in a large district in Florida. Results for the studies are 
available in two CRESST reports (Herman et al., 2015a; Herman et al., 2015b), as well as a 
journal article published by AERA Open (Herman, Epstein, & Leon, 2016). 


Currently, CRESST serves as the independent evaluator for LDC’s federally funded i3 
validation grant. The LDC i3 study is examining the implementation and impact of LDC in two 
large urban school districts: New York City Department of Education and a large school district 
on the West Coast (the latter of which this report is focused on). The evaluation study is a 
comprehensive mixed-methods evaluation to understand the impact of LDC on student learning 
using a quasi-experimental design (QED), as well as to document LDC’s impact on teacher skills 
and practices. Specifically, the evaluation study addresses a wide range of questions about 
program characteristics, conditions, and impacts in the context of two large urban school 


districts. The study draws on data from two cohorts of schools, with each school housing a PLC 
of teachers who engage in professional learning about LDC and implement LDC mini-tasks and 
modules in their classrooms. The study measures teacher implementation and skill 
improvement with teacher surveys, analytic data from LDC’s online CoreTools module building 
platform, and artifact analysis. While we document the core strategies of the LDC model as 
implemented and provide support for LDC improvement, the central focus of our 
comprehensive mixed-methods evaluation is examining the impact of LDC on teacher practices 
and student learning using a quasi-experimental design. 


The first i3 evaluation cohort of schools began implementing LDC during the 2016-2017 
school year, and the second cohort of schools began implementing LDC during the 2017-2018 
school year. This annual progress report examines LDC implementation during the 2017-2018 
school year in a large, West Coast school district. (A parallel progress report focusing on 
implementation in the New York City Department of Education, NYCDOE, will be prepared once 
the student outcome data become available and we finish conducting the analysis.) The current 
annual progress report presents results from (a) surveys of classroom teachers, LDC teacher 
leaders, and school administrators; (b) analyses describing how LDC participants interacted with 
the CoreTools module building platform; (c) rating by CRESST of instructional modules created 
by LDC participants; (d) analysis of the fidelity of implementation across multiple key 
components, indicators, and data sources; and (e) student outcome analyses using the quasi- 
experimental design. 


Survey results provide a window into how LDC was implemented in 2017-2018, the 
perceived utility and effectiveness of various program components, and the perceived impact 
of LDC on both teacher and student skills and knowledge. The analysis of CoreTools user data 
and modules created and adapted by teachers provides evidence on the level of engagement 
with the online platform and module design process, and the quality of the products created by 
the teachers and PLCs. Fidelity of implementation analysis, utilizing the fidelity matrix designed 
collaboratively by LDC and CRESST, provides a broad picture of how schools and the program as 
a whole performed on key fidelity indicators in 2017-2018. Exploratory analyses of the 
effectiveness of LDC in increasing student learning for Cohort 1 schools with two years of LDC 
implementation and Cohort 2 schools with one year of implementation are also included in the 
report. The confirmatory analysis will be conducted next year by pooling teachers from both 
Cohort 1 and 2 schools who implemented LDC in two consecutive years and their students. 


1.1 Logic Model 


The logic model includes four key intervention components that are predicted to be the 
drivers of change in teacher practice and student learning (see Figure 1.1). These components 
include a coach-supported Professional Learning Community formed to implement the LDC 
intervention at the school site and provide a space for teacher collaboration; asynchronous 
support from coaches primarily in the form of feedback in CoreTools through comments and 


peer review; implementation activities completed by participating teachers including module 
development and classroom implementation; and /eadership support at different levels. Note 
that the model also indicates LDC’s expectations for the level of implementation in each area. 


LITERACY DESIGN COLLABORATIVE i3 LOGIC MODEL 


Medium term Long term 


Common Planning 
time for LDC 
Professional 
Learning 
Community (PLC) 
with synchronous 
coach support 


PLC members participate in at least 60 
minutes of common planning time during 
each consecutive two weeks. 

Digital coach joins PLC every other week. 
Sequencing incorporates blocks from LDC 
online courses. 


Asynchronous 


support from LOC Coach provides feedback on modules in 


CoreTools via comments and peer review 
rubric ratings. 


coaches 


Teacher Teachers develop at least one module per 
implementation year in Years 1-2 
activities 


Teachers teach at least two modules per year 
in Years 1-3 


District instructional specialists visit schools 
three times per year to sit inom PLC, meet with 
principal and teacher leader, and observe 
instruction. 

Principals participate in LDC by protecting PLC 
time, setting goals for LOC work, facilitating & 
participating in schoo! visits from district 
instructional specialists, and observing LDC 
instruction. 

Teacher leaders facilitate PLC on the ground 
collaborate with administrator, and 
communicate regularly with coach 


Leadership 
support at 


different levels 


Increased teacher 
expertise and skill 
development 


More effective 
CCRS-aligned 
instruction 
and use of 
formative 


assessment 


Student 
engagement 


Increased 
CCRS skills 


Improved 
test scores 


Higher rates 
of course 
completion 


Students 
acquire 
knowledge and 
skills to be 
college and 
career ready 


Educational 
attainment 
increases 


Higher 
graduation 
rates 


Increased labor 
market 
outcomes 


Contextual Factors: District support, school and student demographics, other policies and initiatives 


Figure 1.1. LDC i3 logic model. 


The logic model predicts that the four key components will lead to increased teacher 
expertise and skill development and more effective Common Core aligned instruction that 
incorporates ongoing formative assessment. In turn, increased teacher capacity and more 
effective instruction will lead to increased student engagement in the short term; increased 
student skill acquisition, higher test scores, and higher rates of course completion in the 
medium term; and improved college and career readiness, education attainment, graduation 
rates, and labor market outcomes in the long term. 


Note that the logic model was revised based on refinements to the program in response 
to learning from both the pilot year (2015-2016) and the first year of Cohort 1’s 


implementation (2016-2017). 


1.2 Evaluation Questions 


Our evaluation questions focus on addressing three main areas: program characteristics 
and implementation, contextual factors and implementation, and program impacts. This 
progress report provides findings on many, but not all, of the evaluation questions. In 
particular, there is limited information available regarding program impacts. This report 
provides an exploratory look at how the refined LDC model is impacting the learning of students 
under Cohort 1 teachers in their second year of implementation and Cohort 2 teachers in their 
first year of implementation. The first confirmatory analysis will be conducted after Year 2 data 
are available for both cohorts of schools/teachers; that analysis will first be shared in next 
year’s annual report. 


1. Program Characteristics and Implementation 


a. 


Who are the participating teachers and schools? Are they representative of the 
teacher/school populations of the respective district on years of teaching, 
education level, prior student performance, etc.? 


How is the LDC program implemented in each district? What are the core 
components (e.g., training, tools, on-site or other direct support) and who are the 
key participants? In what ways did the LDC implementation align with the 
intended model? 


In what ways do teachers implement the LDC tools in their classrooms? To what 
extent do teacher practices align with intended LDC practices? 


How are teachers utilizing the online LDC system (including online tools, 
exemplars, collaborative workspaces, and technical assistance) in terms of 
frequency and use of key features? Does this vary by teacher characteristics? What 
are teachers’ perceptions of the value and quality of the online LDC system? 


What types of LDC professional development opportunities are offered to and 
utilized by teachers at each school/district? Are teachers and schools satisfied with 
the LDC professional development opportunities they received? 


2. Contextual Factors and Implementation 


a. 


What factors facilitate or hinder successful implementation of the LDC model at 
the teacher, school, and district levels? 


How can implementation of the model be improved at the teacher, school, and 
district levels? 


What other educational reforms are being implemented in the participating 
schools and districts? What are their influences on LDC adoption in the schools and 
districts? Are schools able to align reform efforts? 


What are the roles of school and district leadership in shaping the LDC 
implementation? 


3. Program Impacts 


a. 


What is the impact of LDC on the academic performance of participating students 
as measured by the state assessments? 


Do the academic impacts vary by student subgroup including prior achievement, 
race, ethnicity, socioeconomic status, gender, language proficiency, and/or 
disability? Does LDC help close the achievement gap between student subgroups? 


Do the academic impacts vary by student grade level or subject? 


What is the impact of LDC on teacher skill improvement and learning as measured 
by CoreTools and by the quality of LDC modules they produce? What is the self- 
reported impact of LDC on teacher learning? 


To what extent do teachers report changes in their practice (e.g., teaching 
strategy, collaboration with others) and changes in their comfort in implementing 
CCSS during and after the LDC intervention? 


What is the relationship between the fidelity of implementation, fidelity of 
intervention, and student learning? What are the conditions and contexts under 
which LDC tool use is most effective? 


To what extent do Cohort 1 participating schools and teachers continue their LDC- 
influenced practices in the 2019-2020 school year after LDC support ends? What 
contributed to their decision to continue or stop? What factors contributed to 
their levels of continued implementation? How does Cohort 1’s actions align with 
their previously stated intentions for continuation of LDC-influenced practices as 
reported in spring 2017? To what extent do Cohort 2 participating schools and 
teachers plan to continue their LDC-influenced practices after LDC support ends? 


2.0 Study Methodology 


In this chapter, we provide an overview of the methodology behind this look at LDC as 
implemented in 2017-2018. We begin by describing the various instruments and data sources 
for the analyses, including surveys of classroom teachers and teacher leaders participating in 
PLCs and administrators overseeing the implementation; analytic data from LDC’s CoreTools 
platform; module artifacts including samples of student work; LDC administrative data; and 
district administrative data on students and teachers used for outcomes analyses. We then 
describe the sample of educators and schools for each of these data sources. Finally, we discuss 
the methodological approaches for the various analyses we conducted. 


2.1 Data and Instruments 


We next describe each of the data instruments and the elements they contain. Most 
variables are measured at the teacher level, which is the unit at which the LDC intervention is 
being implemented. Administrative data for the analysis of the impact of LDC on student 
learning includes school-, teacher-, and student-level variables. 


Surveys (teachers, teacher leaders, and administrators). In collaboration with LDC, 
CRESST made minor revisions to the 2016-2017 surveys. These refinements involved 
streamlining the language of certain questions, in particular those capturing teacher 
pedagogical practice and the perceived impacts of LDC on teachers and students. In a few 
cases, questions and items were also added to collect systematic information on program 
conditions and impacts that were observed by LDC and CRESST anecdotally. 


In 2017-2018, four different surveys were administered to LDC participants playing three 
different roles: teacher, teacher leader, and administrator. Some teacher leaders were 
classroom teachers who implemented LDC with their students, while other teacher leaders 
were out-of-classroom faculty (coaches or coordinators); these two groups received different 
versions of the survey tailored to their roles. Thus, four versions of the surveys were 
administered in spring 2018: (a) teacher, (b) teacher leader (for coaches and coordinators), 

(c) teacher leader (for teachers), and (d) administrator. 


The surveys were designed to capture multiple perspectives on key aspects of LDC’s logic 
model (see Figure 1.1),4 and to provide data to answer the evaluation’s research questions 
presented earlier. Survey questions targeted at the three roles fall under the domains and 
subdomains in Table 2.1. Domains were selected to align with the LDC i3 logic model and with 
the CRESST evaluation questions. Note that most domains cover multiple subdomains, 
constructs, and survey questions. Professional Learning Community/Teacher Collaboration, for 
example, captures the intensity, frequency, and collaborative environment of common planning 
time; LDC Training and Support includes quality of online courses, utility and effectiveness of 


1The survey domains were aligned to this version of the logic model for the pilot year. The logic model has since 
been revised to align with the revised LDC implementation plan. 


coach support, etc.; and LDC Implementation encompasses module creation, classroom 


implementation of modules, and module peer review. 


Table 2.1 


Survey Domains for Three Respondent Groups 


Teacher 
Domain Teacher leader Administrator 

LDC participation X X X 
Professional learning community/teacher collaboration X X X 
LDC training and support X X X 
LDC implementation 

e Module creation X X 

e Classroom implementation X 

e Module peer review X 
Alignment X X 
Leadership support 

e Teacher leader support X 

e School administrator support/classroom observation X X X 

e Teacher leader leadership role X X X 

e District support X X 
Impact 

e Impact on teacher practice and learning X X 

e Impact on student learning X X 
Scale-up and sustainability X X 
Facilitators and barriers X 
Areas of improvement X X X 


Teachers and administrators were asked to reflect on both LDC’s Impact on Teacher 
Practice and Learning and Impact on Student Learning. Questions within a number of domains 
further asked respondents to reflect on conditions and supports that may potentially impact 
LDC’s implementation. These domains included teachers’ perceptions of Facilitators and 
Barriers to implementation and perceptions regarding leadership roles and support for LDC at 
different levels. Teacher leaders and administrators were also asked for their perceptions 
regarding if and how LDC would be sustained and expanded within the school. Finally, all 
respondents were asked open-ended questions regarding Areas of Improvement for LDC 


implementation. Teacher, teacher leader, and administrator surveys can be found in 
Appendices A, B, and C. 


LDC CoreTools. The CRESST team received the LDC program data on i3 participants’ 
interactions with the CoreTools module-building platform. The data files captured a number of 
behaviors including document page viewing, document editing, document commenting, coach 
use of peer review, exposure to LEARN instructional cycle content, and uploading of student 
work. The data contained date- and time-stamped records of participants’ activities in all of 
these areas. 


Using these data files, we were able to analyze variation in the number of times the 
participants viewed, edited, and commented on documents across the school year. We 
generated descriptive statistics (minimum, maximum, mean, and standard deviation) for the 
number of times participants viewed a document page, edited a module document, and 
commented on a module document. We then produced descriptive statistics on these 
behaviors for each role group (teacher, teacher leader, and administrator), school level 
(elementary, middle, and high), cohort, and content area subgroups. We also compared the 
average engagement in these key activities across two groups: teachers who completed the 
modules we rated in Chapter 5 of this report and those teachers who did not complete a 
module. The results on these overall metrics of engagement are reported in Chapter 4. 


CoreTools data were also used to analyze to what extent (a) teachers were exposed to 
instructional cycles of LEARN content; (b) coaches provided feedback via commenting and peer 
review functions; (c) teachers edited key portions of modules; and (d) teachers uploaded 
student work (a proxy for classroom implementation). Results for these indicators are reported 
in Chapter 6. 


Modules. Our existing module rating rubrics (Reisman, Herman, Luskin, & Epstein, 2013) 
were designed to examine the instructional quality and coherence of the LDC modules, and to 
address the rigor in both content and literacy development materials (i.e., template task, 
student work samples, and descriptions of the pacing and goals of the modules). The first two 
dimensions examined the quality of the teaching task, while the remaining four dimensions 
focused more holistically on module quality: (a) effective writing task; (b) alignment to the CCSS 
and local and state literacy and content standards; (c) fidelity to LDC module instruction; 

(d) quality instructional strategies; (e) coherence and clarity of module; and (f) overall 
impression. Each of the dimensions was rated using a 5-point scale with anchor points on the 
first five dimensions ranging from not present or realized to fully present or realized and the 
final dimension ranging from inadequate to advanced LDC module implementation. This scale 
mimics the structure of the three-level LDC jurying system scale that uses the ordered anchors 
of work in progress, good to go, and exemplary quality. For each dimension, a 1 represented 
the lowest possible level of quality, while a 5 represented the highest level of quality (see 
Appendix D). 


LDC administrative records. The fidelity matrix analyses in Chapter 6 utilize administrative 
records collected by LDC as part of their ongoing program management. These records include 
(a) PLC reflection forms, and (b) attendance records for in-person meetings organized for the 
benefit of school administrators and teacher leaders. The PLC reflection form captures 
attendance at regular PLC meetings, the amount of time spent on LDC during these meetings, 
whether teacher leaders had a separate planning call with coaches, and the subjects that PLCs 
were working on during sessions. 


Administrative data used in student outcomes analysis. Student-level variables utilized in 
the outcome analysis included race/ethnicity, gender, poverty status, special education status, 
English language proficiency, gifted status, grade, and baseline achievement in mathematics 
and ELA and outcome year achievement in ELA on state assessments. Teacher-level indicators 
obtained and utilized included years of teaching experience and teacher attendance. We also 
requested and received roster files that establish a link between teachers and students via 
specific courses. 


2.2 Sample 


Twenty Cohort 1 schools began implementing the LDC program in the 2016-2017 school 
year, with 154 classroom teachers participating and 34 administrators overseeing the work (see 
Table 2.2). The 20 Cohort 1 schools included 11 elementary schools, four middle schools, one 
high school, two K-8 schools, one 6-12 school, and one K-12 school. The program, however, 
experienced a high level of attrition from 2016-2017 to 2017-2018, as summarized in Table 
2.2. Nearly one third of schools dropped out of the program after 2016-2017, and within the 
remaining schools, nearly half of teachers did not continue with LDC in 2017-2018. 


Table 2.2 


Attrition of Cohort 1 Teachers, Administrators, and Schools 


Sample Teachers Administrators Schools 
Participated in LDC in 2016-2017 154 34 20 
Dropped out of LDC after 2016-2017 98 14 6 
Remained in LDC in 2017-2018 56 20 14 
Attrition rate 64% 41% 30% 


To better understand teacher retention rates of LDC schools after the 2016-2017 school 
year, we conducted an interview study of school administrators. Twenty principals and assistant 
principals volunteered to be interviewed. The principal interviews show the diversity of reasons 
for why schools were able to retain teachers, and they also show why teachers left. The main 
reason for teacher retention involved teacher and grade-level team decisions to leave or stay, 
followed by the principals’ decision to switch participation of teachers between the two years. 


Buy-in also seemed to affect teachers’ decisions on whether to continue. See Appendix E for a 
detailed description of the results as well as the methodology for the study. 


A new cohort of schools started in the 2017-2018 school year. Cohort 2 included 31 
schools in total, including 20 elementary schools, eight middle schools, two K-8 schools, and 
one 6-12 school. Because of the high level of attrition in Cohort 1 and the greater number of 
recruited schools in Cohort 2, over two thirds of the participating teachers and administrators 
in 2017-2018 were from Cohort 2. In addition, 34 new participants joined the existing Cohort 1 
schools in 2017-2018. Table 2.3 shows the number of participants in 2017-2018 by cohort 
category and school level. 


Table 2.3 
Number of Participants in 2017-2018 by Cohort and School Level 


Cohort Elementary K-8 Middle 6-12 High Total 
Cohort 1 returning participant 40 0 30 0 6 76 
Cohort 1 new participant 20 0 6 0 8 34 
Cohort 2 participant 154 21 49 16 0 240 
Total 214 21 85 16 14 350 


Table 2.4 
Distribution of Modules Rated by Grade Band, Cohort, and Subject 


Elementary Secondary Total 
Variables # % # % # % 

Cohort 

Cohort 1 returning 12 11.32 15 23.44 15 23.44 

Cohort 1 new 11 10.38 7 10.94 7 10.94 

Cohort 2 83 78.30 42 65.63 42 65.63 
Subject 

ELA 43 40.57 28 43.75 28 43.75 

Science 43 40.57 16 25.00 16 25.00 

Social studies 20 18.87 20 31.25 20 31.25 
Total 106 62.35 64 37.65 64 37.65 


In addition to the CoreTools analytic files, we also received module artifacts from LDC for 
an analysis of the quality of module design. We restricted our analysis to modules that were 
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created during the 2017-2018 school year and included original uploaded student work 
samples. This restriction yielded a pool of 326 modules that were authored or coauthored by 
206 teachers (nearly three quarters of all participating teachers) and one coach, across 42 
schools. Because the number of modules surpassed available resources for rating, we sampled 
the last module created by each teacher wherein they served as the sole or lead author. Our 
intent was to represent teachers’ best LDC work, and presumably, modules created later in the 
school year would be more skilled than those created earlier would. The total sample was 170 
modules. Table 2.4 provides further detail about the distribution of modules. We also 
conducted an exploratory analysis of the seven teachers who created complete modules during 
the 2016-2017 and 2017-2018 school years. 


We now turn to Table 2.5, which displays the overall population and sample sizes for the 
different study measures and analyses included in this report. This includes all participants in 
2017-2018 across the three cohort categories: Cohort 1 returning, Cohort 1 new, and Cohort 2. 
As can be seen, data were available for a large majority of the participants. Ninety-five percent 
of teachers consented to participate in the study, with 84% of all teachers completing the 
survey in spring 2018. The consent rate for administrators (94%) was comparable to that of 
teachers while the survey response rate for administrators (63%) was considerably lower than 
the teacher rate. Nearly all teachers (99%) and administrators (95%) were present in the 
CoreTools dataset, which was provided directly to CRESST by LDC, and did not depend on 
teachers’ individual study consents. A smaller percentage of teachers (78%) were present in the 
PLC reflection attendance data. 


Table 2.5 


Number of Participating Schools and Teachers in the District and Sample Size for Different Instruments 


Instruments Classroom teachers Administrators Schools 

Participated in LDC 286 64 45 
Consented to CRESST study 271 60 45 
Completed survey 246 40 44 
Present in CoreTools dataset 282 61 45 
Authored/coauthored a module with student 206 1 42 
work uploaded 

PLC reflection forms 225 45 45 
Leader in-person meeting attendance N/A N/A 45 
Included in outcomes analysis 148 N/A 36 


The school district required individually signed consent forms before releasing teacher 
data and teacher/student rosters, so for the outcome analysis, we only received data on 
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teachers who consented to participate in the study. As described in detail later, there were 
sufficient data to conduct separate exploratory quasi-experimental analyses of impact for 
Cohort 1 returning middle school teachers, Cohort 2 elementary school teachers, and Cohort 2 
middle school teachers. The final row in Table 2.5 reflects the total number of teachers and 
schools that were represented in these three analyses. There was insufficient student sample 
size to conduct analyses of the impact of Cohort 1 new participants or Cohort 1 returning 
elementary school participants, so the teachers in those groups were not included in analyses. 
The sample was further restricted by the need for student achievement data for both the 
outcome year (2017-2018) and the baseline year (2015-2016). As a result, participants 
teaching either in high school or the primary elementary grades (K-3) were not included in the 
student outcome analyses. Middle school teachers who did not teach a core ELA, science, or 
social studies/history class were also excluded from the analyses. Overall, due to all these 
restrictions, the quasi-experimental analyses represented just over half of the participating LDC 
teachers in 2017-2018. 


2.3 Survey Recruitment and Administration 


As previously noted, roughly one third of the 2017-2018 sample was made up of teachers 
and administrators from Cohort 1 schools, from whom CRESST and LDC had obtained study 
consent in 2016-2017. The recruitment and consent of the new participants was conducted 
during the 2017-2018 school year. This was begun at the beginning of the school year with 
authorized staff attending the LDC Launch Days. Since most LDC participants were unable to 
attend these events, CRESST and LDC also conducted recruitment and the consent process 
through in person contact, video conference, and email. As required by the UCLA Office of the 
Human Research Protection Program, all consent forms included language stating that the 
study was voluntary, that data would be protected, and that by signing the form, participants 
consented to be emailed an electronic survey in spring 2018 and to have CRESST request 
district data linking them and their students. All participating teachers, teacher leaders, and 
administrators were also compensated with a $50 gift card after completing the survey. 


Surveys were administered in spring 2018. CRESST coordinated with LDC staff and coaches 
to administer the online surveys during common planning sessions. Multiple email reminders 
were sent to participants who were not present at these sessions or who otherwise did not 
complete the survey. The teacher survey was closed at the end of the school year in June 2018. 
Administrator surveys were left open until early August when LDC staff met with administrators 
to plan for the next school year. 


2.4 Module Rating Process 


LDC requirements specified that all teachers implement at least two modules over the 
course of the school year, with the first spanning one week based on one text and the second 
spanning across multiple weeks and incorporating multiple texts. Modules could be developed 
as Original units of work or could be adapted from existing modules within the LDC CoreTools 
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Library. Modules could also be either developed or adapted in solo or collaboratively with other 
teachers within their PLC. 


To be included in the analysis, modules were required to include a completed module 
template and at least one student artifact. In total, 326 modules met the above criteria, with 
each teacher authoring and/or coauthoring between one and four modules. As previously 
noted, to align with our research questions and to ensure the feasibility of the rating process, it 
was decided to rate the last module created by a teacher (as a first author) that included the 
required materials. In total, 170 modules were rated with 56 or 32.9% being rated by two 
panelists. Further details about the modules can be found in Appendix F. 


Seven expert raters with experience teaching in the targeted grade spans and content 
areas were recruited from schools in Los Angeles County. Four panels were convened with two 
experts rating each of the following sets of modules: elementary ELA and social science, 
elementary and secondary science, secondary ELA, and secondary social science. It should be 
noted that one rater served on both the secondary ELA and the secondary social science panels. 


Separate trainings lasting approximately two hours were conducted for each panel of 
expert teachers. All trainings were conducted by a member of the evaluation team who is an 
expert on the Common Core and the rating of student and teacher artifacts. The training 
included an overview of the LDC goals, template task, the structure of the modules, and the 
CRESST rating dimensions. Once the training was complete, calibration was conducted by 
having teachers individually rate and then discuss their ratings for one module in the content 
area on which they would be focusing. Rating was then conducted on subsequent days with 
each module individually evaluated. Those modules rated by two expert teachers were then 
discussed with the goal, but not the requirement, to reach consensus (see Carlson & McCaslin, 
2003). All discussions were facilitated by the same evaluation team member who conducted 
the initial training. 


2.5 Analytical Approaches 


Both quantitative and qualitative analytic methodologies were applied to the data to 
answer the evaluation questions about how LDC was implemented, conditions affecting 
implementation, and program impacts. The following describes the approaches used to analyze 
each data set. 


Surveys. Survey responses were analyzed using descriptive statistics for multiple-choice 
items and qualitative coding for open-ended responses. As previously noted, surveys were 
administered to teachers, teacher leaders, and administrators. Some teacher leaders were 
classroom teachers while others were coaches or coordinators. Because of this, the samples 
were not mutually exclusive, with teacher leaders who taught in the classroom represented in 
both the teacher and teacher leader results. Generally, we report the number and percentage 
of respondents who selected different multiple-choice options. In a few cases, where responses 
are numerical rather than categorical, we present means rather than proportions. Descriptive 
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statistics for all multiple-choice items are presented in full in Appendix A for teachers, Appendix 
B for teacher leaders, and Appendix C for administrators. Responses are organized by domain 
and specific questions are identified by question number. For example, we summarize the 
domain of LDC participation in one section, but rely on information from four questions. We 
preface teacher questions with “T,” teacher leader with “TL,” and administrator questions with 
un” 


LDC CoreTools. The first stage of our analysis examined the proportion of all participants 
who created CoreTools user accounts, and engaged in three key behaviors: viewing, editing, 
and commenting on modules. We then analyzed each of the three key measures of 
participants’ interaction with the LDC online tools, and reported frequencies and/or descriptive 
statistics (e.g., minimum, maximum, mean, standard deviation) as measures of participants’ 
engagement with the online LDC system. In addition to reporting the overall results, we provide 
results by cohort, the content areas taught (ELA, social studies/history, and science), by 
participant role (teacher, teacher leader, and administrator), and school level (elementary, K—8, 
middle, 6-12, and high) whenever feasible. Finally, we explored the difference in CoreTools 
engagement between two groups of teachers: those who completed and taught modules and 
those who did not appear to complete the design and implementation process. 


Modules. We used descriptive statistics (means, standard deviations, and percentages) to 
analyze overall and subgroup performance for each content area and school level on each of 
the six dimensions. Additionally, generalizability theory (G theory) was used to examine 
potential sources of error during the rating process to help determine the validity of the scores 
as well as the construct validity of the rubrics (see Shavelson & Webb, 1991). Finally, teacher 
comments during the debriefings were examined to determine other potential issues with the 
rubrics and/or rating process. 


Fidelity of implementation analysis. Fidelity of implementation analysis involves 
descriptive analyses of a variety of data sources, including teacher survey, CoreTools data, PLC 
reflection data, and attendance records for administrator and teacher leader in-person 
meetings. As outlined in the fidelity matrix (see Appendix G), fidelity levels are set at the 
teacher, module, school, and program levels. The process for most indicators involves several 
levels of aggregation, with, for example, module editing being measured at the teacher level, 
school level, and program level. School- and program-level scores are typically based on the 
proportion of teachers or modules that met the threshold for adequate implementation. 


Student outcomes. We employed a quasi-experimental design to examine the effect of 
LDC on the Smarter Balanced Assessment Consortium (Smarter Balanced) ELA assessment 
scores of students in the participating LDC elementary and middle schools in 2017-2018. Before 
conducting the analysis, we used a two-step matching process to identify a reduced pool of 
comparison students and teachers at schools with similar characteristics to the schools in the 
intervention sample. 
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To accomplish this, we first identified the five most similar comparison schools for each 
intervention school based on a Euclidian distance measure, by using the nearest neighbor 
analysis option in SPSS 24.0 (see Fix & Hodges, 1951; Wang, Neskovic, & Cooper, 2007). The 
variables used in this process were the percentage of students eligible for free or reduced price 
lunch, the percentage of African American students, mean baseline student achievement in 
ELA, mean baseline student achievement in mathematics, the average attendance rate of 
teachers, the percentage of teachers with three or fewer years of teaching experience, and the 
school grade span where feasible. We generally used all five identified comparison schools to 
establish the potential matching pool, but for selected LDC schools we only used four of the 
initial five matches. Once the pool of comparison schools was identified, their students and 
teachers were also identified, and student-level matching was conducted so that the resulting 
sample would resemble the type of sample one would expect to obtain through random 
assignment. 


The student-level matching technique we employed was coarsened exact matching (CEM) 
(lacus, King, & Porro, 2011). CEM is a flexible matching approach with many favorable 
properties and allows the researcher to specify the precise conditions under which students are 
matched. For categorical variables, such as race/ethnicity or free or reduced price lunch status, 
this can entail exact matching, while for continuous measures, such as baseline individual 
student achievement and aggregate class-level achievement, cut-points for matching can be 
specified. With this approach we were able to set precise cut-points on the most important 
baseline indicators, such as baseline academic achievement, to ensure that where possible 
every treatment student was matched with a suitable comparison. Student matching variables 
we used in CEM included Hispanic, Black, poverty status, female, English language proficiency 
(English language learner), special education status, gifted status, mean baseline achievement 
in mathematics and ELA, and grade level. 


During matching we also included a few variables capturing information on the teachers 
and peers to whom students were exposed. These variables included mean baseline ELA 
achievement of the student’s peers in his/her core content classes, and the average years of 
teaching experience of the student’s core content teachers. 


The typical structure of course taking at the middle school level involves students being 
exposed to multiple teachers, with each responsible for a different core content class. 
Specifically, middle school students in the study had exposure opportunities across three 
content areas taught by intervention teachers (ELA, science, and social studies/history). As a 
result, students were not necessarily nested under individual teachers, but instead were likely 
to have connections to multiple teachers in the available time period prior to each testing 
outcome (students at the elementary school level were also sometimes exposed to multiple 
teachers, but to a lesser extent). Therefore, LDC effects were estimated using an extension of 
the standard multilevel modeling framework known as multiple membership multiple 
classification (MMMC; Browne, Goldstein, & Rasbash, 2001). 
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Students 


Figure 2.1. Multiple membership multiple classification structure. 


In the MMMC modeling approach, each observation at the lowest level represents one 
student (see Figure 2.1). The weight each teacher receives with respect to each student is 
dependent on the student’s exposure to his or her teachers in each of the three core content 
areas. The total student exposure adds to a unity (i.e., a possible total exposure of 1) across 
their courses in the three content areas in a given school year. While this general weighting 
approach applies to both elementary and middle schools, the course structure of the datasets 
required us to use different weighting procedures in elementary and middle school. 


In both the elementary and middle school datasets, students were linked to teachers 
through statewide course identifiers and accompanying terms in which each student was 
enrolled in a given course under a specified teacher. For elementary students, the terms 
consisted of three marking periods. For middle school students, the terms consisted of two 
semesters. Elementary students were required to have received marks in each of the three 
marking periods, and middle school students were required to have received grades in ELA 
courses in both semesters. Students could potentially have data records connected to multiple 
teachers covering varying time periods in the same course. For each of the three core content 
courses we then collapsed the links into a single measure of number of terms of potential 
content exposure. 


In elementary school, in the event that a student was exposed to more than one teacher, 
each marking period was given equal weight in distributing teacher/student exposure. For 
example, if a student was enrolled in two marking periods under one teacher, then that teacher 
was coded as .67 for having contributed to two thirds of the students’ core curriculum 
exposure. If the same student enrolled in the third marking period with a different teacher, 
then that third marking period teacher would have been coded as .33 and all other teachers in 
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the sample would have been coded as zero. This would then result in the student’s exposure 
adding to a unity (1). 


In middle school, students’ exposure to teachers at the course level in the three core 
content areas was coded in the same manner as in the elementary grades, but based on the 
two semesters. A difference in our middle school coding process was that we did not force each 
core content area into equal weighting. Instead each core content area exposure contributed to 
a core content area total sum that formed the basis from which the weights were proportioned. 
Most commonly, a student had equivalent core instruction exposure in each of the three 
content areas (often two terms each). In that scenario, if a student had exposure to three 
different teachers, then each teacher would contribute one third (.33) of the overall core 
curriculum exposure and all other teachers in the sample would be coded as zero. However, in 
seventh grade the students often only had one semester of science. It was also occasionally the 
case that students would take both a core and an additional elective science course; in these 
cases, both science courses were included in the LDC analysis. As a result, there was more 
variability in the number of units that students accumulated in science than in the other two 
content areas. 


The weighting in middle school was always distributed as a proportion of the total 
semesters across the three content areas. Therefore, if a student accumulated one science unit 
(one semester), two social studies units (two semesters), and two ELA units (two semesters), 
the base number of units would be five. Using that scenario, the science teacher would 
contribute one fifth (.20) of the overall core curriculum exposure with the social studies and 
science teachers contributing two fifths (.40) each, again resulting in the student’s exposure 
adding to a unity. Tabular versions of the above examples can be found in Appendix G. 


For this study, we modeled the treatment intervention variable as a fixed effect at the 
student level in two different ways. The first dosage-dependent approach takes into account 
the students’ level of exposure to the intervention teachers. In this approach, the treatment 
was structured as a continuous response variable, coded as zero for comparison students and 
coded as a positive value for treated students, albeit never exceeding one. The positive value 
assigned to treated students in the dosage-dependent approach was simply the sum of the 
intervention teacher weights linked to the treated student. The second approach was dosage 
independent and classified any student exposed to an intervention teacher via at least one 
course as a treated individual. In this approach the treatment variable was dichotomous: coded 
as one for treated students and zero for comparison students. In the Cohort 2 elementary 
analysis, the two models are very similar because the great majority of the students were 
exposed to just one teacher and, as such, we only present the dosage-independent approach 
for that sample in Chapter 6. 


As with other multilevel models, MMMC accounts for the nonindependence of 
observations within cluster by adjusting the inferences on parameter estimates for the 
correlations between responses in a cluster. This modeling approach, however, becomes 
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computationally cumbersome using traditional frequentist estimation methods. As 
recommended by Browne et al. (2001), to address this issue we employed Bayesian methods 
using Markov chain Monte Carlo (MCMC) techniques. Multilevel models incorporate 
demographic and achievement variables used in the matching design as covariates, making the 
findings “double robust” in that characteristics can be controlled for in both matching and 
outcomes analysis stages. Student demographic and baseline achievement variables that were 
used in the matching process were also included as covariates in the MMMC model. The full 
specifications for both models can be found in Appendix G. 
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3.0 Survey Analysis 


In this chapter we present the survey results. First, we summarize teachers’ responses. 
We then summarize the teacher leaders’ responses, followed by the administrators’ 
responses.* Whenever we felt a comment from an open-ended response might clarify, 
illustrate, or corroborate a finding, we included that comment in the appropriate section. 
Within each of these sections, we organize results by the following domains: LDC participation; 
professional learning community and teacher collaboration; LDC training and support; LDC 
module implementation; leadership support; impact; and issues of scale-up and sustainability 
and facilitators and barriers. The last section summarizes the results about program efficacy 
and improvement, which were completed by all three participant types. We end with a 
summary of results. 


As previously noted, we use acronyms to identify which participants answered specific 
questions for each domain. We preface teacher items with “T,” teacher leader with “TL,” and 
administrator questions with “A.” For example, LDC Participation (T1—4) indicates that teacher 
survey items 1-4 are used to provide information on LDC participation. Survey questions and 
descriptive results are presented in full in Appendix A for teachers, Appendix B for teacher 
leaders, and Appendix C for administrators. 


3.1 Teacher Survey Results 


As noted earlier, 245 teachers spanning 43 schools completed the survey.* Among the 
teachers who completed a survey, the largest proportion taught an elementary school grade 
while the smallest proportion taught at the high school level. More specifically, 63% of teachers 
taught in 28 elementary schools, 6% in two K—8 schools, 23% in 11 middle schools, 5% in one 
grade 6-12 school, and 3% in one high school (see Table 3.1). When examining participation by 
cohort, 45 (18%) were Cohort 1 returning teachers, 30 (12%) were Cohort 1 new teachers, and 
171 (70%) were new teachers in Cohort 2 schools. 


In addition to producing descriptive statistics on the overall sample, we also analyzed 
results separately for elementary and secondary level teachers and by cohort. We highlight 
important differences between these subgroups, when they are apparent. However, because of 
the overlap in membership within the categories, the observed differences must be treated 
with caution. That is, cohort differences were confounded with elementary versus secondary 
comparisons given that Cohort 2 schools were largely at the elementary level during the 
2017-2018 school year. Cohort 1 versus Cohort 2 comparisons also are complicated by the 
presence of a substantial number of Cohort 1 new teachers, whose schools are in their second 
year of implementation but who personally are in their first year of implementation. Further, 


2As noted earlier, some of the teacher leaders were also classroom teachers, coaches, or coordinators. Because of 
this, some teacher leaders were represented in both the teacher leader and classroom teacher samples. 


3One additional teacher started the survey, but did not complete it. This teacher is not included in the analysis. 
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these teachers were sometimes joining PLCs that started in 2016-2017 and sometimes in 
completely reconstituted PLCs in returning schools. Please keep these confounds in mind when 
interpreting results which highlight differences between subgroups. 


Table 3.1 
Number of Schools and Teachers Completing the Survey in 2017-2018 


Cohort 1 returning Cohort1 new Cohort 2 Total 
School type Schools teachers teachers teachers teachers 

Elementary schools 28 24 20 110 154 
K-8 schools 2 0 0 15 15 
Middle schools 11 19 5 34 58 
6-12 schools 1 0 0 12 12 
High schools 1 2 5 0 7 
Total 43 45 30 171 245 


Content Areas Implemented By Secondary Teachers 


50% 
45% 42% 
40% 
35% 
30% 
25% 
20% 
15% 
10% 

5% 

0% 


31% 


18% 


6% 
3% 
[ES = 


ELA (n = 40) History/social studies Science (n = 17) Other (n = 6) Math (n = 3) 
(n = 29) 


Figure 3.1. Secondary teachers’ content areas implemented. 


Among the 245 teachers who completed the survey, 154 reported teaching in an 
elementary school, and 91 teaching in a secondary school. The secondary teachers reported 
teaching one to seven classes (M = 4.6). In these classes, they used LDC materials in zero to 
seven classes (M = 2.7).* As shown in Figure 3.1, the distribution of LDC across content areas 


4One teacher reported not using LDC materials in any classes. 
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was 42% in ELA, 31% in history/social studies, 18% in science, 6% in other areas (e.g., health, 
Spanish), and 3% in mathematics. 


LDC participation (T1—4). Three fourths of responding teachers (75%) reported that 
2017-2018 was their first experience with LDC. While the remaining 61 teachers (25%), who 
were in their second year of implementing LDC, reported teaching approximately two modules 
in 2016-2017 (M = 1.7), there was a wide range of variation in the numbers reported by 
respondents (0-10 modules). These teachers also reported teaching an average of two mini- 
tasks outside of the modules in the prior year, with a range of 0-10. 


Professional learning community and teacher collaboration (T5—9, T31, T38). Almost all 
teachers (96%) participated in a PLC focused at least partly on implementing LDC. The eight 
teachers who had not participated in a PLC reported they had used LDC tools in their planning 
or instruction. All eight of these teachers used CoreTools to access modules; four used 
CoreTools to design modules; two took online LDC courses; and two were given modules by 
other teachers. We hypothesize that given their engagement, these teachers may have partially 
participated in the PLC, perhaps by initially attending meetings and later dropping out. 


The majority of teachers (64%) met with their PLCs every other week. A small percentage 
(12%) reported meeting weekly, and nearly a quarter (25%) reported meeting once a month or 
less. The most common reason cited for not meeting weekly was that PLC members had other 
priorities (54%) and a relative few indicated a lack of protected PLC time (18%). Other less 
frequently cited reasons were limited interest in attending meetings (11%); not a priority of the 
administration (10%); lack of participating teachers (8%); and not enough support from the 
teacher leader (3%). 


The vast majority of teachers reported that LDC PLC meetings lasted 45 minutes or more. 
In addition, 57% reported meetings of 45 minutes to an hour and over a third of teachers (36%) 
reported that meetings lasted longer than an hour. Only 5% of respondents reported meetings 
of less than 45 minutes. About two thirds of the teachers (66%) agreed or strongly agreed that 
their PLCs were given sufficient time to meet. However, in their open-ended responses, 67 
teachers (32% of 194 teacher commenters) indicated that time was a barrier to effective 
implementation. These comments included additional time issues, such as regularity of 
meetings and the need for teachers to have time outside of regular formal meetings to discuss 
and implement LDC. 


Beyond formal PLC meetings, almost half of teachers (46%) said they had informal 
discussions about LDC with their colleagues once a week or more. In addition, 23% of teachers 
reported having informal discussions every other week, 18% once a month, and 13% less than 
once a month. 


Teachers also were positive about the effects of their PLC participation on their 
collaboration with other teachers. Over three quarters (79%) of teachers agreed or strongly 
agreed that after participating in LDC they were more likely to collaborate with other teachers 
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on designing instruction. Almost three quarters (74%) reported that participating in LDC helped 
them develop working relationships with teachers in different grades or subjects. In addition, 
half of teachers (51%) reported sharing their LDC work with colleagues who were not officially 
in LDC. 


LDC training and support (T10-T13). Teachers were asked to evaluate the three primary 
types of training and support offered by LDC: their professional learning community (PLC), 
online course materials, and virtual coach support during and outside of PLCs. 


Teachers found PLCs to be moderately or very effective in multiple ways. As shown in 
Figure 3.2, these include creating an environment in which teachers were comfortable working 
with each other (94%); fostering an environment where teachers shared instructional plans 
with colleagues (87%); allowing space for shared problem solving (83%); and helping teachers 
learn to develop modules (79%). Cohort 1 new teachers were slightly less likely to report that 
their PLC was effective. 


EFFECTIVENESS OF TEACHER'S PLC 


Creating an environment in which teachers were 
comfortable working together. 


Fostering an environment where teachers shared their 
instructional plans with colleagues. 


Allowing space to share student work. 


Helping teachers to improve their LDC instructional plans. 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


B® Not effective/A little effective @ Moderately effective/Very effective 


Figure 3.2. The effectiveness of teacher’s PLC (n = 236). 


The majority of teachers also reported satisfaction with many aspects of the online course 
materials by rating them as good or excellent. As shown in Table 3.2, aspects that were 
endorsed by at least half of the teachers included clarity of information (77%); relevance of 
information (75%); usefulness of resource documents such as the LDC Pitfall Checklist (73%); 
opportunity to extend learning (69%); helpfulness in creating modules (68%); ease of use (54%); 
and usefulness of videos (52%). Cohort 1 new teachers appeared less positive about the 
usefulness of resources (64%), the relevance of the information provided (59%), and their 
helpfulness in creating and/or adapting modules (55%). 
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Table 3.2 
Rating Online Course Materials (n = 236) 


How would you rate each of the following aspects of the 
online course material (in the Learn tab in LDC 


CoreTools) that your coach used or directed you to use? Poor Fair Good Excellent 
Clarity of information presented 3% 20% 58% 19% 
Relevance of information presented 1% 23% 49% 26% 
Ease of use 9% 37% 42% 12% 
Usefulness of resource documents (e.g., LDC Pitfall i . r 0 
Checklist, CCSS Mental Markers, etc.) Pe 20% aoe 20% 
Usefulness of videos 17% 31% 40% 12% 
Degree to which course material helped me to create A% 299% 19% 19% 
and/or adapt LDC modules 

Opportunity to extend learning when needed or desired 5% 26% 47% 22% 


Teachers were also asked to report about the feedback they received. Most teachers 
(96%) felt they were able to get the feedback and support they needed from their LDC coach 
and reported that coaches provided written feedback in a timely manner. In addition, at least 
half the teachers found the following types of asynchronous coaching to be moderately helpful 
or very helpful: individual Zoom conference (68%; 22% did not use); written feedback in 
CoreTools (60%; 21% did not use); and email or phone communication (56%; 32% did not use). 
In general, Cohort 1 returning teachers found the asynchronous coach supports more helpful 
than the other two teacher cohorts. Seventy-three percent of Cohort 1 returning teachers 
found written feedback to be moderately or very helpful, compared to 60% of Cohort 2 
teachers and 42% of Cohort 1 new teachers. New teachers in Cohort 1 again tended to be less 
positive than other teachers in their perceptions of the helpfulness of written feedback (42%) 
or of email or phone communication (48%). 


LDC implementation (T14-29). This domain covers questions on module creation 
(T14-17), module peer review (T27-29), and classroom implementation (T18-26). 


Module creation (T14-17) and module peer review (T27-29). Teachers adapted or 
created two types of LDC instructional products. The mini-tasks are short, generally take one 
class period, and focus on a specific skill. In contrast, modules are longer, more complex units 
composed of a series of mini-tasks, which build to a culminating teaching task, and include a 
culminating writing assignment. 


Using existing LDC templates and exemplars, teachers individually or collaboratively 
adapted zero to four modules during the year. Among the 196 teachers who answered this 
question, most (82%) adapted at least two modules. More specifically, 129 teachers (66%) 
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adapted two modules, 25 teachers (13%) adapted three, and five teachers (3%) adapted four 
modules. In contrast, nine teachers (5%) adapted no modules and 28 teachers (14%) adapted 
one. The mean number of modules that teachers adapted was 1.9. 


Teachers also reported individually or collaboratively creating between zero and 12 
modules from scratch. The majority of teachers (105 or 54%) constructed no original modules, 
50 teachers (26%) created one, 37 teachers (19%) created two, three teachers (2%) created 
three, and one teacher (1%) reported creating 12. The mean number of modules that teachers 
created from scratch was 0.7. 


Teachers also reported how they constructed modules. Most teachers (63%) worked in 
teams of two or more to create modules. A quarter (25%) created modules individually, with 
most of the teachers who did this (41%) teaching at the secondary level. A relative few (18%) 
collaborated with the entire PLC to write modules. 


Less than one third of the teachers (29%) reported attending a Peer Review/Curriculum 
Alignment Workshop during the 2017-2018 school year. Of the 68 teachers who did attend, the 
majority were elementary teachers (53 or 79%). Similarly, a minority of teachers submitted 
their modules for LDC National Peer Review, which was to be expected given the 
preponderance of teachers were new to LDC during the 2017-2018 school year. Of the 66 
teachers who reported submitting modules for peer review, 11 (17%) found the process very 
helpful, 25 (38%) moderately helpful, 18 (27%) a little helpful, and 12 (18%) not helpful. 


When asked to reflect on the instructional strengths of their modules, most teachers 
expressed confidence in their ability to execute all nine of the features of LDC modules that 
were listed (see Table 3.3). More specifically, teachers expressed high confidence in their ability 
to select focus standards and to create a standards-driven writing assignment (92% and 91%, 
respectively). Almost all teachers also felt confident about their ability to make a writing 
assignment relevant and engaging (86%), to identify the skills needed to complete a writing 
assignment (86%), and to select texts and other materials that support deeper learning (81%). 


There were some differences by cohort in these ratings of LDC instructional features. 
Cohort 1 new teachers felt less positive in their ability to execute LDC instructional features 
than the other two teacher groups. For example, Cohort 1 new teachers appeared a little less 
confident in their ability to assess the quality of student writing (59%) or create writing 
assignments that were relevant and engaging for students (76%). 
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Table 3.3 
Rating on Teacher Skills Associated with Creating LDC Modules (n = 236) 


Please indicate to what extent you were able to do Amoderate Agreat 

each of the following when creating LDC modules. Notatall A little bit extent extent 
Select focus standards for a writing assignment 1% 6% 34% 58% 
Create a standards-driven writing assignment 2% 6% 36% 55% 
Select high quality, complex texts and other materials 3% 16% 47% 34% 


that allowed students to engage in deeper learning 


Identify the skills students need to develop to 3% 12% 42% 44% 
complete a writing assignment 


Create daily lessons to teach the skills a student 8% 19% 46% 27% 
needs to complete a writing assignment 


Differentiate instruction by incorporating multiple 6% 22% 47% 25% 
ways of thinking, various levels of complexity, and 
multiple modalities 


Plan for a variety of methods to assess student 8% 21% 46% 25% 
progress (e.g., rubrics and/or mini-task scoring 

guides) 

Assess the quality of writing assignments and/or 6% 21% 42% 31% 


instructional plans using Peer Review/Curriculum 
Alignment Rubric (e.g. Task Pitfalls Checklist, rubric 
indicators) 


Make a writing assignment relevant and engaging for 4% 10% 46% 40% 
students 


Classroom implementation (T18-26). Toward the beginning of the year, teachers were 
asked to Find and Teach a module from the CoreTools Library. The majority of teachers (77%) 
reported that they did this activity. Of those teachers, 80% reported making adjustments to the 
modules they found to adapt them to their classroom needs. 


After implementing the Find and Teach module, teachers were also asked to adapt, refine, 
and/or develop another module. The majority of teachers adapted and refined an existing 
module in the LDC Library (173, 73%) while the remainder (63, 27%) created a module from a 
template in CoreTools. Almost all teachers (93%) reported teaching these modules by the time 
they completed this survey, with seven (3%) reporting that they planned to teach the module 
before the end of the current school year, six (3%) planning to teach it during the next school 
year, and four teachers (2%) stating that they did not have any plans to teach the module. 
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After creating or adapting modules, teachers implemented them in their classrooms and 
reflected on their success. Teachers reported implementing zero to 20 modules (M = 2.2), as 
well as from zero to 20 mini-tasks (M = 4.1) that were not part of modules. Most teachers (213, 
90%) reported implementing at least the recommended two modules over the course of the 
school year. See Table 3.4 for more information. 


Table 3.4 
Number of Modules and Mini-Tasks Implemented (n = 236) 


Modules or mini-tasks Teachers implementing Teachers implementing 
implemented in 2017-2018 modules mini-tasks 
0 4 54 
1 19 30 
2 170 37 
3 31 25 
4 10 14 
5 1 14 
6 0 17 
7 0 2 
8 0 8 
10 0 15 
12 0 8 
15 0 2 
18 0 1 
20 1 9 


Teachers reflected on various dimensions of their module implementation and the 
majority were positive about all aspects. Based on teacher reports, 87% felt that they engaged 
students in accessing complex text to a moderate or great extent; 87% were able to find 
evidence of student performance on targeted standards in student work; 84% engaged 
students in understanding the assignment and the rubric for evaluating student work; 81% used 
evidence of learning to modify instruction; 79% felt they provided feedback to students using 
assignment rubrics; and 79% systematically collected information about student learning. In 
addition, across all six dimensions, between 31% and 43% of the teachers reported that they 
were able to accomplish the activity to a great extent. 
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Few substantial differences emerged between teachers by school level or cohort. A 
notable exception was in using evidence of student progress to modify subsequent instruction 
with Cohort 1 new teachers being less positive (69%) than other cohort groups (80-89% 
positive). 


Leadership support (T33-37). This domain covers questions on teacher leader support 
(T33), administrator support (T34-36), and teacher leadership role in LDC (T37). 


Teacher leader support (T33). Teacher leaders were school staff who provided logistical 
support to the PLCs. Responses indicated that the vast majority of teachers were satisfied or 
very satisfied with the level of teacher leader support. Based on the results, teacher leaders 
effectively supported the PLC meetings (96%); were approachable (96%); effectively invited 
teachers to join LDC (90%); helped teachers align LDC to broader school goals (90%); and gave 
useful feedback (89%). 


School administrator support (T34—36). School administrators were principals, assistant 
principals, or other instructional leaders who observed teachers in action and provided other 
leadership, such as protecting time for PLCs to meet. Teachers provided feedback about the 
support they received from their school administrator. 


The involvement of school administrators, as reported by teachers, was varied. A third of 
teachers (33%) reported that their administrator attended more than three fourths of PLC 
meetings and a similar percentage reported that their administrator attended less than one 
fourth of the meetings. Finally, 14% of teachers reported that their administrators attended 
PLCs about half the time. Secondary teachers reported greater administrator participation than 
did elementary teachers. 


Half of the teachers (50%) reported never being observed by an administrator while 
teaching an LDC mini-task. Despite this, a higher percentage of elementary teachers (55%) than 
secondary teachers (41%) reported having no administrator observations. About one fourth 
(24%) reported being observed once by their administrator; 14% were observed twice; and the 
remaining 11% were observed three or more times. Overall, secondary teachers reported being 
observed more frequently than elementary teachers. 


Almost all teachers (89%) agreed or strongly agreed that their administrator encouraged 
LDC participation. According to teachers, about three fourths of administrators were able to 
allocate resources to ensure PLCs had time to meet (74%); had a firm understanding of LDC 
(73%); and communicated how LDC supported school initiatives and goals (72%). Most 
administrators reportedly also made formative assessment a priority (68%) and used LDC to 
implement standards-driven assignments within existing curriculum (65%). A majority of 
teachers also disagreed that administrators expressed concern that LDC was taking time away 
from other instructional priorities (63%). However, less than half reported that their 
administrator had provided feedback about LDC planning and instruction (45%). 
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Some differences were found for these questions across subgroups. First, secondary 
teachers were more likely than elementary teachers to rate their administrators as supportive 
or effective, and as having a firm understanding of LDC. Second, Cohort 2 teachers (56%) were 
less likely to report that their administrator expressed concern that LDC was taking time away 
from other instructional priorities than Cohort 1 teachers (77%). Third, Cohort 2 teachers were 
also more likely to agree or strongly agree that their school administrators provided feedback 
on LDC planning (49%) than Cohort 1 returning teachers (32%). Finally, 78% of Cohort 2 
teachers agreed or strongly agreed that their school administrators used LDC to implement 
standards-driven assignments with existing curriculum, compared to 52% of Cohort 1 new 
teachers, and 50% of Cohort 1 returning teachers. 


Teacher leadership role (T37). Many teachers who were not playing a formal teacher 
leader role as part of the LDC program nevertheless felt that LDC allowed them to exercise 
leadership in their schools. Most teachers agreed or strongly agreed that LDC helped them set 
instructional goals for LDC work at their schools (62%). About half of teachers felt they were 
able to affect the direction of LDC at their site by having the opportunity to discuss expansion of 
implementation at their school in future years (51%) and work with the teacher leader and 
administrator (51%). In addition, over half of teachers (52%) expressed interest in learning 
more about facilitating LDC implementation at their schools. 


When examining results by subgroup, some differences were found. Sixty-eight percent of 
elementary teachers, compared to 51% of secondary teachers, agreed or strongly agreed that 
they were involved in setting instructional goals for LDC work at their schools, suggesting 
stronger buy-in at elementary versus secondary school levels. When compared to the other 
cohort subgroups, Cohort 1 new teachers tended to be the least positive about their 
involvement in LDC decision making. Sixty percent of Cohort 2 teachers agreed or strongly 
agreed that they were involved in discussions about how to expand LDC at their schools in 
future years, compared to 41% of Cohort 1 returning teachers, and 38% of Cohort 1 new 
teachers. Fifty-three percent of Cohort 1 returning teachers and Cohort 2 teachers agreed or 
strongly agreed that they had the opportunity to work with their LDC teacher leaders and 
administrators to shape implementation, compared to 35% of Cohort 1 new teachers. Lastly, 
54% of Cohort 2 teachers and 53% of Cohort 1 returning teachers agreed or strongly agreed 
that they were interested in learning more about how to lead LDC implementation at their 
schools, compared to 38% of Cohort 1 new teachers. 


LDC impact (T30-32). This domain covers questions on LDC impact on teacher practice 
and learning (T30-31) and student learning (T32). 


Impact on teacher practice and learning (T30-31). We asked teachers about how LDC 
had changed their ability to practice key teacher skills. Specifically, we asked them to focus on 
the change from the beginning to the end of the current school year’s work with LDC. We also 
asked them to provide information about the impact of LDC on their instructional practice. 
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Across all seven items asking about LDC-related pedagogical changes during the year, the 
majority of teachers rated themselves as having improved moderately or a great deal. 
Specifically, teachers perceived improvement in instructional planning by selecting focus 
standards (84%); identifying skills that students needed to complete an assignment (81%); 
creating standards-driven assignments (81%); using evidence of student learning to modify 
instruction (77%); creating daily lessons to teach the skills students needed (76%); collecting 
information on students’ progress (75%); and identifying patterns of student understandings 
and misconceptions (75%). 


Figure 3.3 reports data on other areas of teacher impact. The areas of widest reported 
LDC impact on teacher practice included helping teachers incorporate writing assignments into 
their classroom curriculum (81%); raising expectations for student writing (80%); likelihood of 
teacher collaboration (79%); and incorporating state standards in classroom instruction (77%). 
The least endorsed teacher practice involved sharing LDC work with colleagues outside of the 
LDC PLC (51%). 


IMPACT ON TEACHER PRACTICE AND LEARNING 


Participating in LDC raised my expectations for students' 
writing. 


Using LDC modules became an important part of my 
instructional practice. 


Implementing LDC helped me incorporate my state's 
College and Career-Ready Standards into my instruction. 


LDC helped me incorporate writing assignments into my 
existing curriculum. 


lam more likely to collaborate with other teachers on 
designing instruction after participating in our LDC... 


LDC helped me improve on my teacher evaluation ratings. 


Participating in LDC helped me develop working 
relationships with teachers in different grades and/or... 


| shared my LDC work with colleagues outside of the LDC 
PLC; 
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Figure 3.3. Impact on teacher practice and learning (n = 236). 


When examining the results by subgroup, there were limited differences by cohort and no 
substantial differences by grade level. Despite this a few trends can be highlighted. For 
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example, in general the Cohort 1 returning teachers had the highest ratings and the Cohort 1 
new teachers the lowest. For LDC modules becoming an important part of teachers’ 
instructional practice, 73% of Cohort 1 returning teachers agreed or strongly agreed compared 
to 64% of Cohort 2 teachers, and 55% of Cohort 1 new teachers. For teachers’ sharing their LDC 
work with colleagues outside of the LDC PLC, 59% of Cohort 1 returning teachers agreed or 
strongly agreed that they did, compared to 53% of Cohort 2 teachers, and 28% of Cohort 1 new 
teachers. 


Impact on student learning (T32). Teachers were asked to evaluate the effect of LDC on 
student learning (see Table 3.5). On average, three fourths or more of the teachers agreed or 
strongly agreed that LDC impacted student learning in the following ways: supported students 
to complete writing assignments (82%); increased content knowledge (79%); improved quality 
of writing skills (77%); and increased student capacity to analyze components of the writing 
task (77%). The least endorsed areas of student impact included developing speaking and 
listening skills and reading skills (69%, respectively). 


Table 3.5 
Impact on Student Learning (n = 236) 


Please indicate to what extent LDC had a positive Strongly Strongly 
effect on students in the following areas. disagree Disagree Agree agree 
Reading skills 9% 22% 47% 23% 
Content knowledge 5% 16% 40% 39% 
Ability to complete writing assignments 5% 14% 44% 36% 
Quality of students’ writing 6% 17% 43% 34% 
College and career readiness 9% 19% 43% 29% 
Capacity to analyze and understand the components 6% 17% 43% 34% 
of a writing assignment 
Speaking and listening skills 10% 22% 48% 20% 
Overall literacy performance 7% 19% 49% 25% 
Performance on assessments throughout the school 9% 21% 50% 20% 
year 


When examining the results by subgroup, some cohort differences emerged. For example, 
Cohort 1 new teachers again appeared as the least positive and Cohort 1 returning teachers 
(those who had actually had two years of LDC experience) generally the highest. Some cohort 
differences to note for the following effects on student learning (moderately to a great deal) 
include reading skills (75% Cohort 1 returning, 70% Cohort 2 teachers, and 55% Cohort 1 new); 


30 


quality of students’ writing (84% Cohort 1 returning, 79% Cohort 2 teachers, and 69% Cohort 1 
new); college and career ready skills (82% Cohort 1 returning, 72% Cohort 2 teachers, and 55% 
of Cohort 1 new); overall literacy performance (82% Cohort 1 returning, 75% Cohort 2 teachers, 
and 59% Cohort 1 new); and performance on assessments throughout the year (72% Cohort 2 
teachers, 70% Cohort 1 returning, and 55% Cohort 1 new). 


Facilitators and barriers (T38). Successful implementation of LDC depends on a number of 
factors. We asked teachers to weigh in on the effect of these factors on implementation (see 
Figure 3.4). Most teachers (87%) agreed or strongly agreed that their school had adequate 
technology to support teachers’ use of LDC, although in open-ended responses, some teachers 
expressed concern about lack of technological knowledge which affected their access to 
CoreTools and the Zoom meetings. Almost three fourths of teachers felt adequately prepared 
to implement modules in the classroom (73%) and thought it was easy to find and adapt LDC 
mini-tasks (73%). However, most teachers (59%) found it challenging to find content-rich texts 
to use with LDC modules. Most teachers (66%) felt their PLC was given sufficient time to meet, 
although the majority of teachers cited time as a barrier in the open-ended responses. 


FACILITATORS AND BARRIERS TO IMPLEMENTATION 


My LDC PLC was given sufficient time to meet. 


| felt adequately prepared to effectively implement LDC 
modules in my classroom. 


It was challenging to find content-rich reading materials 
for the LDC modules | developed. 


My school had adequate technology to support teachers' 
use of LDC. 


It was easy to find and adapt LDC mini-tasks for use in my 
classroom. 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


mw Disagree/Strongly disagree m Agree/Strongly agree 
Figure 3.4. Facilitators and barriers to implementation (n = 236). 


There were some differences by cohort but not school level in how teachers responded to 
facilitators and barriers in LDC implementation. In general, Cohort 1 new teachers again 
appeared the least positive. Although 70% of Cohort 2 teachers agreed or strongly agreed that 
their LDC PLC was given sufficient time to meet, Cohort 1 teachers, both returning (57%) and 
new (55%) were less likely to agree. Eighty-four percent of Cohort 1 returning teachers agreed 
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or strongly agreed that they felt adequately prepared to effectively implement LDC modules in 
their classrooms, compared to 79% of Cohort 2 teachers and 59% of Cohort 1 new teachers. 


In terms of finding content-rich materials for the LDC modules teachers developed, most 
teachers found it challenging: 64% of Cohort 1 teachers, 50% of Cohort 1 returning teachers, 
and 48% Cohort 1 new teachers. Lastly, 91% of Cohort 1 returning teachers expressed interest 
or strong interest in learning more about how to lead LDC implementation at their schools, 
compared to 89% of Cohort 2 teachers and 72% of Cohort 1 new teachers. 


3.2 Teacher Leader Survey Results 


This section presents the survey results from teacher leaders. At each school, the teacher 
leader coordinated PLC meetings with LDC coaches. Teacher leaders were either classroom 
teachers or coaches/coordinators. A total of 48 teacher leaders responded to the survey.° 


LDC participation (TL1-—2). The majority of the 48 teacher leaders who responded to the 
survey (33, 69%) reported that they were new to LDC and did not have experience with LDC 
prior to the 2017-2018 school year. Of the 15 teacher leaders (31%) who had prior experience 
with LDC, they reported teaching between 0-7 (M = 1.7) LDC modules and between 0-10 
(M = 1.7) LDC mini-tasks outside of the modules. 


Professional learning community and teacher collaboration (TL3-—6). Similar to teacher 
responses, the majority of teacher leaders (73%) reported that their PLCs met every other 
week. Less than a quarter (17%) of teacher leaders reported that their PLCs met once a week or 
more frequently. The 40 teacher leaders whose PLCs did not meet weekly reported that the 
primary barriers were that PLC members had other competing priorities (50%) and that PLC 
time was not protected (23%). 


Also similar to teacher responses, 60% of teacher leaders said these meetings lasted 45 to 
59 minutes. Over a third (35%) reported that meetings lasted 60 to 74 minutes. A small 
minority (4%) said meetings were less than 45 minutes. Nearly a third (31%) reported that they 
had informal discussions about LDC with teachers every other week, and over a quarter (27%) 
reported that they had informal meetings once a week. 


LDC training and support (TL7—12). Teacher leaders evaluated the effectiveness of the 
various training and support methods provided during LDC implementation. These included PLC 
meetings, online course material, and coach support. Their responses are very similar to the 
overall teacher ratings. 


Teacher leaders overwhelmingly reported that the PLC meetings were effective. All 
teacher leaders (100%) felt that the LDC PLC was moderately to very effective in creating an 
environment in which teachers were comfortable working with each other. Teacher leaders 


*Forty-eight teacher leaders started the survey, but one teacher leader stopped after item TL6. Forty-seven 
teacher leaders completed the full survey. 


32 


also reported that their PLC was effective in each of the following ways: fostered an 
environment where teachers shared instructional plans with colleagues (94%); allowed space to 
share student work (92%); and helped teachers learn to improve their LDC instructional plans 
(89%). 


Online course materials used by the coaches during PLC meetings also received high 
ratings from the teacher leaders, slightly higher than overall teacher ratings. Most teacher 
leaders (85%) rated the clarity of information as good or excellent; 85% for relevance of 
information; 85% for helpfulness to teachers in creating modules; 83% for usefulness of 
resource documents; 83% for opportunity to extend learning when needed or desired; and 75% 
for usefulness of videos. A little over half of the teacher leaders (55%) thought that the online 
course materials were easy to use. 


LDC coaches also were highly regarded by teacher leaders and their responses showed 
more frequent interaction with the coach than teacher peers. All (100%) reported that they 
were able to get the feedback and support they needed from LDC coaches, and 98% reported 
that the coaches provided written feedback on modules in a timely manner. Different types of 
asynchronous coach support were rated by the majority of teacher leaders as moderately or 
very helpful. Teacher leaders found the support, when used, helpful. These supports were email 
or phone communication (98% helpful; 2% did not use), individual video conferencing with 
coaches (96% helpful; 2% did not use), and written feedback in CoreTools (83% helpful; 13% did 
not use). Additionally, teacher leaders were asked about LDC coach support outside of the PLCs. 
All teacher leaders (100%) reported that their LDC coaches connected them with additional 
resources when needed. Almost all teacher leaders (98%) agreed or strongly agreed that their 
coach responded quickly; 96% thought their coach was easy to work with; 94% said they were 
able to reach their coach with questions; 94% said the coach was knowledgeable and provided 
high quality guidance; and only 36% of teacher leaders reported that it was challenging to 
structure PLC time with the coaches. 


Teacher leaders had the opportunity to attend LDC professional development meetings, 
both in person and online (these were in addition to PLC meetings). The average number of 
meetings attended was 5.9, with a range of 0 to 20. With respect to LDC support outside of 
coaches, 89% of teacher leaders agreed that LDC offered sufficient professional development 
opportunities for them to lead the initiative at their schools, and 85% agreed that there was 
adequate technical support for CoreTools’ issues. 


Module creation (TL13-16). Teacher leaders reported the ways in which modules were 
created in their PLCs and judged the extent to which they were able to embed targeted 
instructional practices while creating the modules. 


Teacher leaders were asked how many existing LDC modules were adapted by their PLCs. 
Answers ranged from 0 to 10, with a mean of 2.5. Teacher leaders were also asked how many 
modules their PLCs created from scratch, either individually or as a group, and they reported O- 
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6 modules, with a mean of 1.0. Modules were most commonly created via a collaborative 
process, either by teams of two or more teachers (57%) or by the PLC as a whole (23%). Some 
teachers (21%) created the modules individually. 


In general, teacher leaders were extremely confident in their ability to carry out targeted 
instructional practices while creating their modules; their responses are slightly more positive 
than those of general teacher participants. Almost all teacher leaders (98%) felt they were able, 
to a moderate or great extent, to select focus standards for the writing assignment; 98% 
created a standards-driven writing assignment; 92% made writing assignments relevant and 
engaging to students; 89% identified the skills students needed to complete the writing 
assignment; 89% were able to select high-quality texts and other materials; 81% differentiated 
and provided multiple opportunities for students to engage with the assignment; 81% assessed 
quality of writing assignments or instructional plans; 79% created daily lessons to teach the 
necessary skills; and 77% planned for a variety of ways to assess student progress. 


Impact on student learning (TL17). Teacher leaders were asked to evaluate the effect of 
LDC on student learning and here again their responses tend to be more positive than those of 
teacher participants. On average, over three quarters of teacher leaders agreed or strongly 
agreed that LDC impacted student learning in the following ways: improved quality of writing 
skills (92%); supported students to complete writing assignments (89%); increased content 
knowledge (89%); increased student capacity to analyze components of the writing task (87%); 
improved overall literacy performance (83%); developed skills needed for college and career 
readiness (83%); increased performance on assessments throughout the year (83%); developed 
speaking and listening skills (79%); and developed reading skills (79%). 


Alignment (TL21). Teacher leaders were asked how LDC aligned with other instructional 
priorities and programs at their schools. Alignment was generally perceived to be quite high as 
reported in Figure 3.5. Nearly all (98%) agreed or strongly agreed that LDC was a strategy for 
implementing statewide college and career-ready standards; 89% thought LDC helped prepare 
students for state assessments; 87% said LDC helped teachers create writing assignments to 
use within their existing curricula; 77% viewed LDC as complementing other initiatives at the 
school; and 60% reported their school connected LDC to specific school goals. Notably, 
however, about half reported that time spent on LDC interfered with other initiatives, and 
almost three quarters (81%) believed that it was difficult for teachers to focus on LDC because 
of other competing priorities at the school. 
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LDC ALIGNMENT 


Our school connected LDC implementation to our specific 
schoolwide goals. 


LDC helped teachers create writing assignments to use 
within their current curricula. 


LDC complemented other initiatives taking place in my 
school. 


| view LDC as a strategy for implementing my state's 
College and Career Ready Standards. 


The time spent implementing LDC interfered with other 
important initiatives at my school. 


LDC helped prepare students in my school for current 
state assessments. 


It was difficult for teachers to focus on LDC because of 
other competing priorities at the school. 


Our instructional leaders are using LDC to implement 
standards-driven assignments within the existing... 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


mw Disagree/Strongly disagree m Agree/Strongly agree 
Figure 3.5. LDC alignment (n = 47). 


Teacher leader role (TL20). Teacher leaders reflected on their leadership role in LDC 
implementation at their school sites. Overall, results were very positive and indicated that the 
majority of teacher leaders felt engaged by and invested in the LDC implementation at their 
school. Almost all (96%) agreed or strongly agreed that they met regularly with their LDC coach 
to manage the work plan; 94% felt they were able to build capacity as an instructional leader as 
a result of their LDC leadership role; 83% were involved in planning how to meet teacher 
learning needs by differentiating LDC implementation; 77% were involved in discussions about 
how to expand LDC implementation in future years; 70% believed their role as an LDC teacher 
leader helped them advocate for additional resources on their campus; 72% were involved in 
adjusting LDC problems of practice at their school; 68% met regularly with their school 
administrator; and 51% were confident that they could lead their PLC in the future without the 
assistance of an LDC coach. 


Leadership support (TL18-19, T23). This domain covers questions on school administrator 
support (TL18-19) and district support (TL23). 


School administrator support (TL18—19). Similar to teachers’ responses, teacher leaders 
were asked to report on school administrators’ involvement with LDC. About one third (32%) 


reported that their administrators attended more than three fourths of the PLC meetings; 30% 
less than one fourth; 17% about half; 15% about one fourth; and 6% about three fourths. 


Teacher leaders were also asked to report on the types of support that they received from 
their school administrator(s). A large majority of teacher leaders (89%) reported that their 
administrators encouraged teachers to participate; 77% reported that resources were allocated 
to ensure that the LDC team could meet; 75% said their administrators had a firm 
understanding of LDC; 72% said their administrator communicated how LDC supported other 
school goals and initiatives; 66% reported their administrators made formative assessment a 
priority at their school; 64% said administrators used LDC to implement assignments with 
existing curriculum; and 51% said they received feedback about their LDC planning or 
instruction from their administrator. Notably, only 17% of teacher leaders reported that their 
administrators expressed concern that LDC was taking time away from other instructional 
priorities, which is slightly less than that reported by teacher participants. 


District support (TL23). Teacher leaders appeared slightly less positive about the district’s 
support of LDC than were teachers. Over half of teacher leaders (53%) agreed or strongly 
agreed that district leaders supported implementation of LDC; 43% said that district 
professional development efforts aligned with LDC; 36% reported that district leaders visited 
their school to discuss LDC implementation; 34% agreed that district leaders were interested in 
implementing LDC at other schools; and 34% thought district leaders had a firm understanding 
of LDC. 


Scale-up and sustainability (TL22). Teacher leaders were also asked their perceptions 
about the scale-up and sustainability of LDC implementation. As shown in Figure 3.6, more than 
two thirds of the respondents agreed or strongly agreed that they expected most teachers to 
continue with LDC the following year (70%) and felt that teachers and administrators were 
committed to sustaining the initiative (70%). About half felt that collaboration was taking place 
across subject areas and/or grade levels. Furthermore, one quarter agreed or strongly agreed 
that teachers who were not participating in the PLC were using LDC, and half (51%) believed 
that participation would increase during the next school year. Since scale-up was not a 
particular goal for LDC implementation and given that a majority of respondents were in Cohort 
2 schools, the reports of the teacher leaders that LDC is being used outside the PLCs should be 
viewed as a promising trend regarding the future use of LDC. 
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SCALE-UP AND SUSTAINABILITY 


| expect that most teachers participating in LDC this year 
will continue to do so next year. 


Teachers at my school who were not part of the LDC PLC 
meetings used to planning process and/or LDC CoreTools. 


As a result of LDC, new collaborations across grades 
and/or subjects were created or are being launched at... 


Teachers and administrators at my school are committed 
to sustaining the LDC initiative. 


| expect our LDC PLC to increase in size next year. 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


mw Disagree/Strongly disagree m Agree/Strongly agree 
Figure 3.6. Scale-up and sustainability (n = 47). 


3.3 Administrator Survey Results 
This section presents the survey results from school principals and assistant principals. 


LDC participation (A1) and professional learning community (A2). Of the 34 school 
administrators who responded to this survey, 24 (71%) were principals and six (18%) were 
assistant principals, and four (12%) responded other. The majority of administrators reported 
attending more than half of the LDC PLC sessions, with 21% attending less than one fourth, 18% 
attending about one fourth, and 27% attending about a half. Another 8% reported attending 
three fourths, and 27% attended more than three fourths. Overall, respondents to the 
administrator survey reported somewhat higher attendance than did teacher respondents. One 
explanation may be that given less than two thirds of administrators responded to the survey, 
the respondents may be a more engaged group than the population of all administrators 
overseeing LDC. 


Training and support (A3—4). Administrators had the opportunity to attend professional 
development sessions, either online or in person, such as Launch Day and administrator 
meetings. The range of meeting attendance was between zero and 20, with an average of 
attendance of five meetings (MM = 5.3). 


Perceptions of LDC were overwhelmingly positive. All administrators thought their school 
had adequate technology to access LDC. Almost all administrators agreed or strongly agreed 
that LDC offered sufficient professional development for teacher leaders (97%); there were 
sufficient professional development opportunities for administrators (97%); that they were able 
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to reach LDC staff with questions (94%); and that LDC staff were able to connect them to 
additional resources (88%). 


Classroom observation (A5—6). Administrators varied concerning the number of 
observations that they reported conducting during the school year. Half of respondents 
indicated that they observed one or two times, about one third (35%) reported observing three 
or more times per year, and 15% reported that they did not observe LDC implementation at all. 
These reports appear somewhat more positive than those from teacher surveys. Furthermore, 
most administrators (86%) who observed LDC instruction believed that LDC modules were 
moderately or very effective in developing students’ literacy skills. 


Impact on teacher practice (A7). The majority of administrators observed significant 
improvement in all areas of teacher practice probed on. More specifically, most respondents 
(94%) believed teachers had improved moderately or a great deal in selecting focus standards; 
82% in identifying necessary skills to complete the writing assignment; 79% in creating 
standards-driven writing tasks; and 79% in creating daily lessons to teach skills to complete the 
writing task. In addition, 71% of administrators believed that teachers improved their use of 
evidence of student performance to shape instructional decisions; 65% in identifying patterns 
of student understandings or misconceptions; and 62% in collecting information on student 
progress in a systematic way. 


Impact on student learning (A8). Administrators were positive about the effects of LDC 
on students. More than three fourths of respondents agreed or strongly agreed that LDC 
improved students’ ability to complete writing assignments (82%), writing quality (79%), overall 
literacy performance (79%), content knowledge (79%), and understanding of the components 
of the writing task (79%). Most administrators also felt that LDC participation supported 
students’ development of college and career-ready skills (71%), speaking and listening skills 
(71%), and reading skills (68%). Many administrators (62%) also believed that participation 
resulted in increased performance on assessments throughout the school year. 


Administrator leadership role (A9). Almost all administrators felt they played an active 
role in LDC implementation. More specifically, more than three fourths of respondents agreed 
or strongly agreed that they allocated resources to ensure LDC team could meet (91%); made 
changes to school schedules to accommodate LDC PLC time (88%); and were able to shape LDC 
implementation at their schools (82%). Large percentages of administrators also reported that 
they met regularly with the LDC teacher leader (74%); led discussions about how to expand LDC 
implementation in future years (71%); and were involved in discussions about differentiating 
LDC implementation to meet teacher needs (68%). 


Alignment (A10). Administrators were asked to reflect on how well LDC aligned with 
other school initiatives, programs, and curricula (see Figure 3.7). Respondents generally felt 
that LDC aligned well with their school. All administrators agreed or strongly agreed that LDC 
helped teachers create writing assignments to use within current curricula and it 
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complemented other initiatives at the school. In addition, almost all administrators indicated 
that they were using LDC to implement standards-driven assignments with existing curriculum 
(94%), they considered it a strategy for implementing state college and career-ready standards 
(91%), and it helped students prepare for state assessments (91%). The only areas where less 
than half of administrators agreed or strongly agreed involved teacher evaluation ratings (47%) 
and whether time spent on LDC interfered with the implementation of other school initiatives 
(32%). 


LDC ALIGNMENT 


Our school connected LDC implementation to our specific 
schoolwide goals. 


LDC helped teachers create writing assignments to use 
within their current curricula. 


LDC complemented other initiatives taking place in my 
school. 


| view LDC as a strategy for implementing my state's 
College and Career-Ready Standards. 


The time spent implementing LDC interferred with other 
important initiatives at my school. 


LDC helped prepare students in my school for current 
state assessments. 


It was difficult for teachers to focus on LDC because of 
other competing priorities at the school. 


| am using LDC to implement standards-driven 
assignments within the existing curriculum. 


LDC helped improve teacher evaluation ratings 
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Figure 3.7. LDC alignment (n = 34). 


Scale-up and sustainability (A11). The outlook for LDC implementation in future years 
was positive according to administrators. As shown in Figure 3.8, 91% of respondents indicated 
that teachers and administrators at their school were committed to sustaining LDC, and all felt 
that most teachers would continue implementing during the 2018-2019 school year. The 
majority of administrators also noted cross-grade and/or subject collaborations (68%) and that 
they expected their PLC to increase in size for the next school year. A small percentage of 
respondents (18%) also noted that teachers who were not currently part of the PLC were using 


the LDC planning process and/or CoreTools, which might be the source of their confidence 
about the growth of the program for the next year. 


SCALE-UP AND SUSTAINABILITY 


| expect that most teachers participating in LDC this year 
will continue to do so next year. 


Teachers at my school who were not part of the LDC PLC 
meetings used the LDC planning process and/or LDC 
CoreTools. 


As a result of LDC, new collaborations across grades 
and/or subjects were created or are being launched at my 
school. 


Teachers and administrators at my school are committed 
to sustaining the LDC initiative. 


| expect our LDC PLC to increase in size next year. 
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Figure 3.8. Scale-up and sustainability (n = 34). One respondent selected don’t know for the second, 
fourth, and fifth items. 


District support (A12). Administrator ratings of district support appeared less positive 
than did their ratings in other areas and suggest an area of need. The only two items where at 
least half of respondents agreed or strongly agreed involved district leader support of the 
implementation of LDC (65%) and alignment between district professional development efforts 
and LDC (50%). In contrast, only 47% of respondents reported that district leaders visited the 
school to discuss LDC implementation, 38% agreed that district leaders were interested in 
expanding LDC to other schools, and 38% thought district leaders had a firm understanding of 
LDC. 


3.4 Open-Ended Responses for All Participants 


Each survey respondent was asked for feedback in response to three questions: (a) What 
supports did you find the most useful and why? (b) What supports were not helpful and why? 
and (c) In what ways could LDC implementation be improved in your school in the future? The 
following section summarizes the findings for the responses to these questions from 194 
teachers, 40 teacher leaders, and 34 administrators. Results for these questions are presented 
thematically. We begin by discussing some of the general results after which we discuss the 
responses related to the PLCs, virtual and then in-person coaching, and finally the CoreTools. 


General views. In general, responses to the survey questions were positive. For example, 
68 teachers (35%), 21 teacher leaders (53%), and 23 administrators (68%) said that everything 
was helpful. Furthermore, only six teachers and one teacher leader commented that no part of 
the intervention was helpful. 


Professional learning community. Many of the survey respondents noted positive views 
of the LDC PLC meetings. More specifically, there were 160 positive comments, with 111 
specifically noting the participants’ appreciation of the virtual meetings with their LDC coaches. 
Respondents also appreciated the opportunity to collaborate with other teachers (29 
comments) as part of their PLC. 


Despite the positive views, respondents did note areas for improvement. The most 
common suggestion for improving implementation in the future was to provide more or 
protected time for LDC. Specific themes found about time (77 comments) included requests for 
meeting more frequently, more time and support to both plan and implement modules (not 
necessarily within the PLC time), to be paid for their time, and to use time more efficiently. 
While teachers were generally positive about the content of the PLCs, they expressed some 
dissatisfaction about logistical issues involving technology use, the scheduling of meetings, the 
pacing of meetings, and assignments, among others. 


While less predominant, other suggestions were made about how to improve 
implementation within the PLCs. At least 17 respondents suggested an earlier start or more 
teacher buy-in to get the implementation off the ground at their school. Small numbers of 
respondents specifically requested summer training or full days of training, as opposed to an 
hour or two. There were also 29 comments, almost all from teachers, who felt that increased 
teacher buy-in was necessary in order for LDC to succeed at their sites. Ideas for accomplishing 
this buy-in included offering compensation, having coverage for substitutes (for planning, LDC 
meetings, and classroom visits), and providing additional materials. 


Respondents also had some competing ideas concerning participation. For example, some 
teachers who were given no choice about participation felt that teacher buy-in would increase 
if participation was optional. This contrasts with a few administrator comments in which they 
stated that LDC should be a requirement and that it shouldn’t be on a voluntary basis. Related 
to this issue of teacher buy-in, a few teacher leaders observed that PLC meetings would be 
more productive if teachers were committed to the program. In addition, 14 respondents 
suggested including more teachers, nine suggested making LDC a schoolwide program, and a 
few respondents suggested that all ELA teachers be included or that there should be separate 
PLCs for the different grade levels rather than cross-grade PLCs. 


Coach (virtual). LDC coaches were frequently mentioned, with 111 comments that 
coaches were one of the most useful supports available and 28 comments that the coaches 
were not helpful. These included comments about Zoom videoconference meetings with the 
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entire PLC, individual Zoom meetings with coaches, email contact, and written feedback via 
CoreTools. 


Positive comments covered a range of supports. Respondents praised coaches’ feedback 
and guidance in navigating LDC website content like CoreTools, the library of modules, and the 
online courses. Respondents appreciated that they could ask questions and receive immediate 
feedback. They also appreciated one-on-one videoconferences with coaches, written feedback 
via CoreTools, and email. 


Respondents who felt virtual coaches were not helpful cited technical problems as the 
most common issue. Across all groups, there were 13 mentions of technology being a problem, 
and one positive comment about technology. When the videoconferencing software and 
hardware did not function properly, participants felt there was time wasted. Other negative 
comments included that the virtual coaches were ineffective, with some teachers mentioning 
that having a virtual coach did not allow for personalized or differentiated support during the 
PLC meetings. A handful of respondents just felt uncomfortable with the medium. Notably, 
among suggestions for improvement, 29 comments were made about requesting more in- 
person coach visits, while only one participant requested more video conferences. 


Coach (in-person). Here we report on comments about several types of in-person 
coaching, including virtual coaches’ site visits and assistance from the district liaison and LDC 
liaison. A few teachers also mentioned support from teacher leaders in the context of coaching. 
There were 51 comments indicating that in-person coaching was one of the most useful 
supports, and 17 comments about how helpful teacher leaders were to teachers. There were 29 
requests for more in-person coaching, with some suggesting that having a coach observe 
classroom implementation of LDC modules and then providing feedback and immediate 
debriefing would be helpful. 


Whether in person or virtually, coaching was the most mentioned support in the open- 
ended responses. Respondents appreciated receiving feedback in any format, whether via 
individual Zoom conferences, during PLC conferences, via email, or in the CoreTools platform, 
and respondents also requested more coaching. 


CoreTools. While comments were balanced, there were more positive ones (125) than 
negative ones or those that focused on suggestions (75). Included in this were comments about 
the content of the LDC library, most of which were positive. Participants who liked CoreTools 
mentioned the wealth of information on the site, including the rubrics and accessing mini-tasks 
and modules. Online courses, part of the CoreTools platform, received 15 positive mentions as 
well. 


For those participants who struggled with CoreTools, most cited navigation issues (18), 
library modules not being user friendly (7), and limited content in the library modules (10). 
Related to this idea of limited content were 33 comments that the LDC modules in the library 
did not meet students’ needs, either in reading level, content (e.g., elementary grades, science), 
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or English learner status and students with disabilities. The three most commonly cited barriers 
to mastering use of CoreTools were ease of use or navigation, lack of time to explore, and 


difficulty modifying existing modules. Related to the idea of ease of use, the most popular 


suggestion with 12 comments called for additional training in navigating CoreTools. 


3.5 Summary of Results 


This survey captured the responses of 82% of all participants in the district’s second year 
of implementation. In addition to answering closed-ended questions, most of the respondents 


also provided narrative comments about aspects of the program they felt were useful as well as 


those that could use improvement. Generally, respondents provided positive feedback. Overall, 
the survey results suggest the following. 


Overall findings 


Across all participant groups, survey responses showed positive attitudes toward LDC. 
All measures of satisfaction or improvement were rated positively by a majority of 
participants. 


LDC training and support 


Teacher leaders were almost universally reported to be highly approachable, effective, 
and knowledgeable. 


LDC coaches received almost unanimous positive feedback, with 93% of teachers and 
98% of teacher leaders reporting that their coaches gave them appropriate and timely 
feedback and support. A small number of participants submitted negative comments 
regarding coaches. 


While 87% of teachers agreed that their school had adequate technology to support 
LDC implementation, open-ended responses indicated that successfully conducting 
Zoom meetings was a common complaint. 


Professional learning community and teacher collaboration 


Almost two thirds of teachers felt they had sufficient time to meet during professional 
learning community sessions and about three fourths felt that their administrators 
protected allocated resources to ensure that they could meet. However, the most 
frequently requested modification for future years was for more time during and 
outside of PLCs to plan modules, implement, and receive feedback about 
implementation, including having pay and substitute coverage. 


Almost two thirds of teachers and three fourths of teacher leaders reported meeting 

in their PLCs once every two weeks, the frequency expected by LDC. For teachers, the 
most cited barrier to meeting every week was that PLC members had other priorities 

that competed with LDC participation, and for teacher leaders, the most cited barrier 
was that PLC time was not protected. 
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A little over half the teachers expressed interest in learning more about how to lead 
LDC implementation at their schools in the following year. The majority of teacher 
leaders (70%) and all administrators (100%) expected their teachers to continue with 
LDC the following year. 


LDC implementation 


The majority of teachers (71% to 92%) reported success in nine key areas of LDC 
module development. Teachers were most confident in selecting focus standards, 
creating the writing assignment, identifying skills needed in the module, and making 
writing assignments relevant and engaging, and somewhat less confident in skills 
involving lesson planning and formative assessment. 


With regard to their classroom implementation of LDC modules, the majority of 
teachers reported success with all six key areas queried (79% to 87%). Teachers were 
most confident with engaging students in complex text, locating evidence of standards 
in final student work, and engaging students in understanding the assignment and 
rubric. 


Leadership support 


Almost all teachers reported that their administrators encouraged LDC participation at 
the school, though only 45% of teachers reported receiving feedback from 
administrators about LDC and 50% reported never being observed while teaching an 
LDC task. Administrators’ recollections about their classroom observations were more 
positive regarding their frequency. 


Impact on teachers and students 


The majority of teachers reported improving in skills related to their own practice and 
learning in all seven skills listed (75% to 84%). Teachers were most likely to report 
impact in the following skills: selecting focus standards, creating standards-driven 
writing assignments, and identifying skills students need in writing assignments. 


Approximately 80% of teachers agreed that participating in LDC raised their 
expectations for student’s writing, helped them incorporate writing assignments into 
their existing curriculum, and made them more likely to collaborate with other 
teachers on designing instruction. 


Three fourths of teachers, 83% of teacher leaders, and 79% of administrators agreed 
that LDC helped improve students’ literacy performance. The two most highly rated 
areas of impact, according to all three groups, were students’ ability to complete 
writing assignments and quality of students’ writing. 


Facilitators and barriers 


Most teachers found CoreTools to be helpful, but almost half of teachers reported 
that the website was not easy to use and that videos were not useful. The majority of 
teachers thought that online course materials were clear, relevant, and useful (i.e., 
checklists, mental markers). 
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This year’s survey responses were consistently positive across all dimensions and 
participant groups. Suggestions and recommendations provided in this report were primarily 
about adapting to and ameliorating district constraints, such as protected time for meetings 
and technological limitations. 
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4.0 Analyses of LDC CoreTools Data 


In the following section, we report on how participants interacted with LDC’s CoreTools 
online system. We begin by presenting participation rates for key CoreTools activities, including 
creating a user account, viewing modules, editing modules, and commenting on modules. We 
then dig deeper into CoreTools viewing, editing, and commenting by sharing descriptive 
statistics for all i3 CoreTools users followed by results broken down by key subgroups (including 
LDC role, school level, cohort, and content area taught). After that we compare the level of 
engagement for i3 CoreTools users who completed and taught a full-length module to those 
users who did not complete and teach a module during the school year. Finally, we summarize 
results. 


4.1 CoreTools Activity Participation Rates 


The four key CoreTools activities we examined are creating a user account, viewing 
modules, editing modules, and commenting on modules a. Among the 343 CoreTools users, 282 
were classroom teachers (with 43 of those individuals acting as teacher leaders), 9 were 
coaches or coordinators playing the teacher leader role, and 52 of them were administrators, 
with some individuals in each category acting as teacher leaders. As seen in Table 4.1, nearly all 
participants used CoreTools to at least some degree. Ninety-eight percent of all participants 
created a user account, 91% of participants viewed modules, and 77% edited modules. 
Commenting on modules was a less common activity with only 27% of all participants doing so. 


Table 4.1 


CoreTools Feature Participation Rates: 2017-2018 (Counts and Percentages) 


User Viewed a Edited a Commented 

Participant type n account module module ona module 
Teacher 243 239 (98%) 235 (97%) 210 (86%) 62 (26%) 
Teacher leader (classroom teacher) 43 43 (100%) 43 (100%) 42 (98%) 25 (58%) 
Total teachers 286 282 (99%) 278 (97%) 252 (88%) 87 (30%) 
Teacher leader (coach/coordinator) 9 9 (100%) 9 (100%) 8 (89%) 3 (33%) 
Administrator (principals and 55 52 (95%) 32 (58%) 11 (20%) 5 (9%) 
assistant principals) 
Total participants 350 343 (98%) 319 (91%) 271 (77%) 95 (27%) 


Table 4.1 also displays subgroup results for participants playing different roles in the i3 
implementation. These subgroups mirror the four groups to whom we administered surveys: 
teachers, teacher leaders who teach in the classroom, teacher leaders who are coaches and 
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coordinators, and administrators (principals and assistant principals). Teachers and teacher 
leaders had higher participation rates than the principals and assistant principals, as expected. 


Nearly all participants created a CoreTools user account. Significantly, even 
administrators, who generally were not a part of the regular PLC meetings, overwhelmingly 
created user accounts. This suggests that the vast majority of administrators overseeing LDC 
PLCs had at least some familiarity with the online platform. Likewise, nearly all PLC teachers 
and teacher leaders viewed modules in CoreTools. In addition, almost two thirds of principals 
and assistant principals viewed modules in the platform. 


We consider editing modules and mini-tasks to be the key indicator of deep engagement 
with the CoreTools module building platform. Eighty-eight percent of participating teachers 
edited at least one module or mini-task. Not surprisingly, principals and assistant principals 
were much less likely than teachers to edit materials in CoreTools with just one out of five 
doing so. 


Overall, adding comments to modules was a much less common activity. A little less than 
one third of classroom teachers commented and only eight coaches, coordinators, or 
administrators did so. Commenting was not surprisingly most common among teacher leaders, 
whose role was most conducive to giving feedback to PLC members. 


4.2 Engagement With Key CoreTools Activities 


In this section, we describe participants’ level of engagement with three key CoreTools 
activities: viewing modules, editing modules, and commenting on modules. Descriptive 
statistics are reported for all participants, as well as a number of subgroups. Those subgroups 
capture the role the individual played in LDC (teacher, teacher leader, and administrator), the 
level of the school at which the participant worked (elementary, K—8, middle, 6-12, or high), 
the cohort of schools the participant belonged to (Cohort 1 returning, Cohort 1 new, or Cohort 
2), and in the case of teachers, the content area taught (elementary/multiple subjects, 
secondary ELA, secondary history/social studies, and secondary science). As noted earlier, some 
participants played multiple roles in the intervention, so the teacher and teacher leader groups 
overlap to some degree as they do in our survey analysis. Descriptive statistics are only 
reported for groups with five or more members. The samples for the viewing, editing, and 
commenting analyses are the 343 LDC participants with CoreTools user accounts in 2017-2018 
(see Table 4.1). 


Module viewing. In Table 4.2 we present descriptive statistics on how many times i3 
participants viewed modules, both overall and by subgroup. We present the minimum and 
maximum number of views, means and standard deviations, and the sum of total views across 
participants. Overall, the average participant viewed modules or mini-tasks over 41 times, 
although the range varied greatly from zero to 276 views. There were 24 participants with user 
accounts who did not view any modules (about 7% of the overall sample). The standard 
deviation of 43.8 also shows the variability in views, suggesting that about two thirds of all 
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participants viewed modules between zero and 85 times, with the remaining one third viewing 
modules an even greater number of times. 


Table 4.2 


Descriptive Statistics for the Number of Times a Participant Viewed a Module in CoreTools, by Participant 
Subgroup 


Subgroup n Min Max M SD Sum 


All participants 343 0 276 41.1 43.8 14,094 


Participant role 


All teachers 282 0 276 47.7 44.8 13,446 
All teacher leaders 52 3 276 75.0 57.1 3,900 
All administrators 52 0 29 5.9 8.1 307 
Cohort 
Cohort 1 returning teachers 75 0 276 38.7 51.1 2,904 
Cohort 1 new teachers 33 0 93 36.1 29.7 1,190 
Cohort 2 teachers 235 0 272 42.6 43.0 10,000 
School level 
Elementary school participants 210 0 272 44.7 46.3 9,383 
K—8 participants 21 0 199 52.0 48.2 1,091 
Middle school participants 82 0 135 32.3 29.5 2,652 
6-12 participants 16 0 65 34.9 20.7 559 
High school participants 14 0 276 29.2 72.7 409 


Content area taught 


Elementary/multiple subjects 173 0 272 53.2 47.2 9,205 
Secondary ELA 40 1 93 38.3 26.5 1,270 
Secondary social studies/history 27 3 93 42.4 26.7 1,144 
Secondary science 19 8 135 36.3 33.6 689 


As would be expected, teacher leaders on average had the greatest number of views 
among the three participant roles, while administrators viewed the least number of times. 
There was considerable variation in viewing behavior across participant subgroups within these 
categories. Cohort 1 returning teachers who were in their second year of implementation, 
Cohort 1 new teachers who were in their first year of implementation, and participants in 
Cohort 2 all viewed modules at similar levels on average. Furthermore, participants in 
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elementary and K—-8 schools viewed considerably more modules on average than their middle 
and high school peers. 


Module editing. As shown in Table 4.3, on average participants who engaged in editing 
modules did so 12 times over the course of the school year. There was a wide range of 
engagement from editing zero times to making 136 edits to modules and mini-tasks. Seventy- 
two participants with CoreTools user accounts (21%) did not do any editing of modules. Two 
thirds of participants edited between zero and 28 times. 


Table 4.3 


Descriptive Statistics for the Number of Times a Participant Edited a Module in CoreTools, by 
Participant Subgroup 


Subgroup n Min Max M SD Sum 


All participants 343 0 136 12.4 16.1 4,235 


Participant role 


All teachers 282 0 136 14.3 16.8 4,041 
All teacher leaders 52 0 136 29.0 25.5 1,507 
All administrators 52 0 16 1.5 3.7 79 
Cohort 
Cohort 1 returning teachers 75 0 136 14.1 22.4 1,055 
Cohort 1 new teachers 33 0 46 7.9 9.1 262 
Cohort 2 teachers 235 0 65 12.4 14.3 2,918 
School level 
Elementary school participants 210 0 136 11.7 15.4 2,457 
K-8 participants 21 0 65 20.8 20.3 437 
Middle school participants 82 0 65 12.1 15.2 993 
6-12 participants 16 0 35 12.4 10.0 198 
High school participants 14 0 101 10.7 26.6 150 
Content area taught 
Elementary/multiple subjects 173 0 136 14.1 16.5 2,444 
Secondary ELA 40 0 65 12.4 14.9 414 
Secondary social studies/history 27 0 65 17.7 17.1 477 
Secondary science 19 0 65 14.0 17.4 265 
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As with page viewing results, teacher leaders edited more than teachers not playing a 
leadership role, and administrators edited much less than teachers. Cohort 1 returning teachers 
and Cohort 2 teachers engaged in editing at considerably higher levels on average than 
teachers newly joining Cohort 1 PLCs. There were no substantial differences in editing behavior 
across school level and content area taught. 


Table 4.4 


Descriptive Statistics for the Number of Times a Participant Commented on a Module in 
CoreTools, by Participant Subgroup 


Subgroup n Min Max M SD Sum 


All participants 343 0 22 0.9 2.5 21 


Participant role 


All teachers 282 0 22 1.0 2.6 292 
All teacher leaders 52 0 22 2.2 4.0 116 
All administrators 52 0 2 0.1 0.4 6 
Cohort 
Cohort 1 returning teachers 75 0 14 0.6 1.9 48 
Cohort 1 new teachers 33 0 2 0.3 0.6 10 
Cohort 2 teachers 235 0 22 1.1 2.7 250 
School level 
Elementary school participants 210 0 8 0.7 1.5 153 
K—-8 participants 21 0 16 2.2 4.0 47 
Middle school participants 82 0 22 1.1 3.5 85 
6-12 participants 16 0 2 0.2 0.5 3 
High school participants 14 0 14 1.3 3.7 18 
CONTENT AREA TAUGHT 
Elementary/multiple subjects 173 0 16 1.0 2.0 179 
Secondary ELA 40 0 22 1.0 3.6 39 
Secondary social studies/history 27 0 20 1.2 4.0 31 
Secondary science 19 0 10 0.7 2.3 14 


Module commenting. Only 95 participants, representing less than one third of all 
participants, made at least one comment on a module. As shown in Table 4.4, while 
participants commented between one and 22 times, the average was only one. Across 
participant roles, teacher leaders had the highest level of engagement, with over two 


50 


comments on average. As with editing, participants who newly joined existing PLCs showed a 
lower level of engagement than returning teachers or teachers in Cohort 2 schools. 


4.3 CoreTools Engagement as an Implementation Variable 


To evaluate the validity of CoreTools engagement as an indicator of LDC implementation, 
we examined the relationship between the three CoreTools engagement measures and module 
implementation. As described in the next chapter, CRESST identified full-length modules that i3 
teachers created and/or adapted and taught in their classrooms. While all teachers may have 
taught the module(s) in their classrooms, we took the presence of uploaded student work as 
evidence of teachers having done so. This parameter yielded a sample of 209 modules, 
associated with 282 LDC teachers. Figure 4.1 displays the mean number of CoreTools views, 
edits, and comments for the 204 participants who completed and taught full-length LDC 
modules and their 78 peers for whom we do not have evidence of full module implementation. 
As shown, teachers who completed and taught full-length LDC modules exhibited considerably 
more engagement with CoreTools than their peers, across all three metrics. The results suggest 
that participants who engage deeply with the module building platform are more likely to 
complete and teach LDC modules. 


Mean CoreTools Views, Edits, and Comments Made by Teachers 


60.0 
54.1 
50.0 
40.0 
31.0 
30.0 
20.0 15.6 
11.6 
10.0 
1.2 0.7 
Number of CoreTools views Number of CoreTools edits Number of CoreTools comments 
™ Completed and taught modules (n = 204) @ Did not complete and teach modules (n = 78) 


Figure 4.1. Mean number of CoreTools views, edits, and comments made by teachers who did and did 
not complete and teach modules. 


4.4 Summary of Results 


Descriptive analysis of CoreTools user behavior shows broad engagement with LDC’s 
module building platform but the depth of that engagement varied greatly across users. Nearly 
all 2017-2018 LDC participants, including principals, created a CoreTools user account, and the 
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vast majority of teachers in PLCs viewed and edited modules. The average teacher, however, 
only made 14 edits to modules within CoreTools. While this might be interpreted as evidence 
that teachers’ engagement in instructional design was somewhat limited, the CoreTools data do 
not provide any measures of instructional design work conducted offline by individuals or 
groups of teachers. 


Furthermore, this descriptive analysis provides a broad rather than deep look, and does 
not take into account LDC expectations that teachers will do very little design work on their first 
module and extensive instructional design on their later modules. Teacher leaders on average 
engaged at higher levels than their peers not playing a leadership role in the LDC intervention. 
Teachers newly joining Cohort 1 School PLCs in their second year of implementation engaged at 
lower levels than did Cohort 1 returning teachers and teachers in the Cohort 2 schools. This 
finding suggests it may be challenging for teachers joining PLCs after a year to get up to speed, 
or perhaps that these new PLC members may be less motivated than PLC members who joined 
the program earlier. 
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5.0 Module Artifact Analysis 


This section presents results for the analysis of modules submitted during the 2017-2018 
school year. We first present overall results across the grade bands. This is followed by separate 
results for the elementary and secondary modules. Finally, we present results of an exploratory 
analysis examining results across time for Cohort 1 teachers who submitted modules across the 
last two school years, the qualitative results, and a summary of the findings. Appendix F has the 
generalizability theory study results. 


Table 5.1 shows the overall means and standard deviations by grade band for the module 
artifact analysis. When examining the ratings, modules at the elementary level received higher 
mean ratings on all dimensions than did modules at the secondary level. More specifically, 
mean ratings for the elementary modules ranged from 3.92 to 4.35, indicating that on average 
all dimensions were moderately to sufficiently present or realized. In contrast, mean ratings for 
the secondary modules were all in the moderately present or realized range (3.44 to 3.77). 


Furthermore, as one would hope to find, the average summary scores for the first five 
dimensions were similar to the overall impression ratings (Dimension 6) provided by the expert 
teachers. Similar results were found when examining the frequencies, with between 68% and 
84% of elementary modules and between 48% and 70% of secondary modules receiving ratings 
of 4 or greater out of a possible 5 on six dimensions. Greater detail about the frequencies can 
be found in Appendix F. 


Table 5.1 
Means and Standard Deviations for the Modules 


Elementary Secondary Overall 
Dimension (n = 106) (n = 64) (n=170) 
Dimensions 

1. Effective writing task 4.35 (0.77) 3.45 (1.27) 4.01 (1.08) 
2. Standards alignment 4.03 (0.89) 3.44 (0.96) 3.81 (0.96) 
3. Fidelity to LDC instruction 4.16 (1.06) 3.64 (1.03) 3.96 (1.08) 
4. Quality instructional strategies 4.09 (1.05) 3.77 (1.04) 3.97 (1.05) 
5. Coherence/clarity of module 3.92 (1.03) 3.75 (1.14) 3.86 (1.07) 
6. Overall impression 3.94 (1.06) 3.69 (0.89) 3.85 (1.00) 
Average (Dimensions 1 to 5) 4.11 (0.78) 3.61 (0.75) 3.92 (0.80) 
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5.1 Analysis of Elementary Modules 


The following section presents descriptive results for the elementary modules 
disaggregated by content area and then by i3 cohort grouping. Results from the generalizability 
theory modules as well as the expanded descriptive results for the elementary modules can be 
found in Appendix F. 


Descriptive results by content area. Table 5.2 presents descriptive results for the 
elementary modules by content area. A few comparative observations can be made. First, when 
looking across the content areas, modules generally received higher ratings on the effective 
writing task (Dimension 1). Second, the dimensions where modules received somewhat lower 
means varied depending upon the content area. More specifically, means were highest for the 
science modules with the exception of Dimension 2, which focused on standards alignment. 
Furthermore, the ELA modules received the highest mean rating for Dimension 2. Third, in 
comparing the Dimension 6 ratings to the average for summary scores for the first five 
dimensions, it is apparent that the science raters were more closely calibrated in their overall 
impressions then were the ELA and social studies raters. 


Table 5.2 


Means and Standard Deviations for the Elementary Modules by Content Area 


ELA Science Social studies 
Dimension (n = 43) (n = 43) (n = 20) 

Dimensions 

1. Effective writing task 4.19 (0.79) 4.54 (0.74) 4.30 (0.73) 

2. Standards alignment 4.16 (0.69) 3.93 (1.12) 3.95 (0.69) 

3. Fidelity to LDC instruction 3.86 (1.23) 4.47 (0.91) 4.05 (0.89) 

4. Quality instructional strategies 3.86 (1.01) 4.56 (0.88) 3.60 (1.10) 

5. Coherence/clarity of module 3.93 (1.03) 4.02 (1.12) 3.70 (0.80) 

6. Overall impression 3.77 (1.13) 4.30 (0.86) 3.55 (1.10) 
Average (Dimensions 1 to 5) 4.00 (0.83) 4.30 (0.76) 3.93 (0.64) 


Finally, when examining the frequencies, science modules also tended to have the highest 
proportion of ratings of 4 or greater out of a possible 5 (see Appendix F). The only exceptions 
involved Dimension 5, with ELA and science both having 70% of modules reach this threshold, 
and Dimension 2 with ELA and social studies both having greater proportions (84%). 


Descriptive results by cohort. Table 5.3 presents descriptive results for the elementary 
modules by i3 cohort grouping: Cohort 1 returning, Cohort 1 new, and Cohort 2. In this case, 
ratings were lowest for the modules submitted by Cohort 1 returning teachers, with means 
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ranging from 3.17 to 3.75 depending upon the dimension. In contrast, means were 3.99 or 
greater on all dimensions for the modules submitted by new Cohort 1 teachers and Cohort 2 
teachers. Furthermore, in comparing the Dimension 6 ratings to the average for the first five 
dimensions, it is apparent that expert teachers’ overall impressions were fairly aligned for the 
Cohort 1 new and Cohort 2 modules, but were conservative for the modules submitted by 
returning Cohort 1 teachers. Finally, the proportion of ratings of 4 or greater were generally 
lowest for Cohort 1 returning modules and highest for the Cohort 2 modules (see Appendix F). 
Despite this, because of the large variation in the sample sizes these results should be 
considered tentative. 


Table 5.3 
Means and Standard Deviations for the Elementary Modules by i3 Cohort Grouping 


Cohort 1 returning Cohort 1 new Cohort 2 
Dimension (n=12) (n= 11) (n = 83) 

Dimensions 
1. Effective writing task 3.67 (0.49) 4.36 (0.92) 4.45 (0.74) 
2. Standards alignment 3.75 (0.97) 4.27 (0.90) 4.04 (0.88) 
3. Fidelity to LDC instruction 3.75 (1.06) 4.00(1.61) 4.22 (0.99) 
4. Quality instructional strategies 3.25 (1.14) 4.18 (1.33) 4.21 (0.95) 
5. Coherence/clarity of module 3.33 (1.07) 4.09 (1.38) 3.99 (0.96) 
6. Overall impression 3.17 (1.11) 4.18 (1.25) 4.02 (0.99) 
Average (Dimensions 1 to 5) 3.57 (0.75) 4.18 (1.08) 4.18 (0.71) 


5.2 Analysis of Secondary Modules 


The following section presents descriptive results for the secondary modules 
disaggregated by content area and then by i3 cohort grouping. Results from the generalizability 
theory modules as well as the expanded descriptive results for the secondary modules can be 
found in Appendix F. 


Descriptive results by content area. Table 5.4 presents descriptive results for the 
secondary modules by content area. While the subgroups are small, some general observations 
can still be made. For example, the mean ratings for the ELA and social studies modules were 
highest for Dimension 4 to Dimension 6, which assessed the quality of instructional strategies, 
coherence and clarity of the module, and overall quality. In contrast, the science modules 
received the highest ratings for Dimension 1 and Dimension 3, which measured the effective 
writing task and fidelity to LDC instruction. In addition, the science modules had lower means 
than did those for the other content areas on all dimensions except the effective writing task. 
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Furthermore, as one would hope to find, the average summary scores for the first five 
dimensions were similar to the overall impression ratings (Dimension 6) provided by the expert 
teachers. Finally, for the ELA and social studies modules the proportion of ratings of 4 or 
greater out of a possible 5 were generally greatest for Dimensions 4 through 6 (see Appendix F). 


Table 5.4 
Means and Standard Deviations for Secondary Modules by Content Area 


ELA Science Social studies 
Dimension (n = 28) (n = 16) (n = 20) 

Dimensions 

1. Effective writing task 3.54 (1.20) 3.56 (0.89) 3.35 (1.63) 

2. Standards alignment 3.54 (0.92) 3.25 (1.24) 3.55 (0.83) 

3. Fidelity to LDC instruction 3.68 (1.22) 3.56 (1.03) 3.65 (0.75) 

4. Quality instructional strategies 4.00 (1.15) 3.38 (1.09) 3.70 (0.66) 

5. Coherence/clarity of module 4.11 (0.92) 3.13 (1.31) 3.75 (1.12) 

6. Overall impression 3.86 (0.93) 3.31 (1.08) 3.75 (0.55) 
Average (Dimensions 1 to 5) 3.77 (0.80) 3.32 (0.81) 3.61 (0.58) 


Results by cohort. Table 5.5 presents descriptive results for the secondary modules by i3 
cohort grouping: Cohort 1 returning, Cohort 1 new, and Cohort 2. In this case, ratings were 
highest for the modules submitted by new Cohort 1 participants, with means ranging from 4.00 
to 4.71 depending upon the dimension. In contrast, all means were less than four for the 
modules submitted by Cohort 1 returning and Cohort 2 teachers. Modules in the different 
cohorts also had different weaknesses with means lowest for Dimension 3 for the returning 
Cohort 1 teachers, Dimension 5 for the new Cohort 1 teachers, and Dimension 1 for Cohort 2 
teachers. Furthermore, as with the content area results, the average summary scores for the 
first five dimensions were similar to the overall impression ratings provided by the expert 
teachers. Finally, as shown in Appendix F, the proportion of ratings that were four or greater 
were generally highest for Dimensions 5 and 6, which measured coherence and clarity of the 
module and overall impression. 
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Table 5.5 
Means and Standard Deviations for the Secondary Modules by i3 Cohort Grouping 


Cohort 1 returning Cohort 1 new Cohort 2 
Dimension (n = 15) (n=7) (n = 42) 
Dimensions 

1. Effective writing task 3.67 (1.23) 4.71 (0.49) 3.21 (1.26) 
2. Standards alignment 3.80 (0.94) 4.14 (0.69) 3.24 (0.96) 
3. Fidelity to LDC instruction 3.40 (1.06) 4.14 (0.90) 3.64 (1.03) 
4. Quality instructional strategies 3.87 (1.06) 4.14 (0.69) 3.64 (1.06) 
5. Coherence/clarity of module 3.87 (1.19) 4.00 (1.83) 3.67 (1.00) 
6. Overall impression 3.87 (0.83) 4.29 (0.49) 3.52 (0.92) 
Average (Dimensions 1 to 5) 3.72 (0.72) 4.20 (0.55) 3.47 (0.76) 


5.3 Exploratory Analysis of Modules 


As previously noted, in order to examine potential growth among the Cohort 1 teachers 
who continued with LDC in 2017-2018 (returning teachers), ratings were compared for 
teachers who submitted complete modules in both the 2016—2017 and 2017—2018 school 
years. Table 5.6 presents descriptive results for the last complete module submitted by these 
teachers. Additional descriptive results for the exploratory analyses can be found in Appendix F. 


Table 5.6 
Means and Standard Deviations for the Exploratory Analysis of Elementary Modules 


2016-2017 2017-2018 Change 
Dimension (n=7) (n=7) (n= 14) 
Dimensions 

1. Effective writing task 3.43 (1.40) 3.86 (0.38) 0.43 (1.27) 
2. Standards alignment 3.00 (1.00) 4.14 (0.69) 0.43 (1.40) 
3. Fidelity to LDC instruction 3.43 (1.40) 4.14 (1.07) 0.43 (1.27) 
4. Quality instructional strategies 3.43 (1.40) 3.14 (1.46) -0.71 (0.76) 
5. Coherence/clarity of module 3.86 (0.90) 3.71 (1.11) -0.43 (0.79) 
6. Overall impression 3.43 (1.40) 3.14 (1.46) -0.43 (1.13) 
Average (Dimensions 1 to 5) 3.43 (1.09) 3.80 (0.80) 0.03 (0.88) 
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Table 5.6 presents results for the seven elementary teachers who submitted complete 
modules during both school years. While the sample sizes are too small to be significant, a few 
observations can be made. First, when looking at the two time points, mean ratings for 
Dimensions 1 to 3 were greater during the 2017—2018 school year while means for the 
remaining three dimensions were greater during the previous year. When examining the 
average summary scores for the first five dimensions, means were the same as for Dimension 6 
at the first time point, but expert teachers seemed to provide more conservative ratings of 
overall impression for the second time point. 


5.4 Qualitative Results 


Expert raters were asked to participate in debriefings in an attempt to shed light on their 
ratings and suggest potential ways to improve ratings in the future. These debriefings were 
conducted at the end of each week of ratings. The following presents key findings concerning 
Dimensions 1 through 5, as well as our expert raters’ overall impressions. 


Dimension 1: Effective writing task. While a majority or the modules received higher 
ratings of 4or 5 on this dimension, the writing task was not always realized. Modules that 
received lower ratings for this category tended to have a writing task that was either overly 
generic or too verbose for the expert rater to understand how it would be taught. In addition, 
the student background section was not used in a consistent manner, with some teachers using 
it to provide background about their students and others using this component as a script of 
what to say. Finally, the extension option was rarely used and tended to provide little detail. 


Dimension 2: Standards alignment. Elementary teachers were more consistent than 
secondary teachers in their specification of standards for the modules. When providing lower 
ratings for this dimension, the expert raters noted the following issues. First, modules tended to 
have reading standards, but were less likely to have focused writing standards to support the 
essays being written by the students. Second, some of the adapted modules included multiple- 
grade standards that did not align with the grade level listed. Because of the limited 
backgrounds and reflections provided it was difficult for the expert raters to determine whether 
this was done purposefully or not. Third, some of the adapted modules included standards from 
states other than California. 


Dimension 3: Fidelity to LDC module instruction. Expert raters noted some overarching 
trends regarding modules that received higher versus lower levels of fidelity. First, those 
modules with stronger fidelity tended to have stronger standards alignment. Second, modules 
that showed lower fidelity tended to be missing sufficient transition to writing standards and 
mini-tasks. Third, many modules included a mini-task on writing body paragraphs but failed to 
provide instruction on how to write introductory and closing paragraphs. 


Dimension 4: Quality instructional strategies. Raters noted two trends that impacted 
their ratings of the instructional strategies. First, mini-tasks were not always well organized, 
making it difficult to determine what was being taught on a daily basis. Second, multiple issues 
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with student work were encountered. This included attachments that were hand written and 
difficult to read as well as work that appeared to be from a previous administration of the 
module. Expert raters also noted that it was easier to measure quality when teachers included 
the completed rubrics with the student work samples. 


Dimension 5: Coherence and clarity of module. The expert raters found the student work 
to be vital to measuring the coherence and clarity of the modules. This was especially true for 
modules that included multiple work samples at varying levels as well as those that included 
pre-writing and not just final papers. 


Dimension 6: Overall impressions. Feedback on the overall quality of the modules tended 
to mimic the ratings of coherence and clarity. Modules that received higher ratings on 
Dimension 5 also received higher ratings on Dimension 6. In part, this had to do with the rater’s 
ability to discern how well the module was actually realized as evidenced in the student work 
samples, student background, and teacher reflection sections. In addition, modules that 
seemed ready to be implemented as written tended to be considered higher in overall quality. 


5.5 Summary of Results 


Based on the primary analyses, modules submitted by teachers during the 2017-2018 
school year were more successful than those submitted during the previous two years of the 
study. When examining the overall samples, mean ratings were in the high 3s to mid 4s for the 
elementary modules and in the mid- to high 3s for the secondary modules. In addition, when 
looking at the proportion of ratings, the majority of modules tended to receive high ratings of 4 
or 5 on the dimensions, indicating that a module was sufficiently present or realized or fully 
present or realized. The only exception involved Dimension 2, involving standards alignment, 
for the secondary sample with only 48% of modules meeting this threshold. 


Modules submitted by the elementary and secondary teachers also tended to have 
different strengths. More specifically, elementary teachers generally performed better in how 
they set up the modules (Dimension 1 to Dimension 3) while the secondary teachers generally 
performed better in how they carried out the instruction (Dimension 4 to Dimension 6). While 
there were a few exceptions, when examining the descriptive results for the elementary 
subgroups, means for the dimensions were generally highest for the science modules and for 
the modules submitted by new Cohort 1 teachers or by Cohort 2 teachers. In contrast, for the 
secondary subgroups the means were highest for ELA and new Cohort 1 teachers. Because of 
the variation in sample sizes, though, this result should be considered tentative. 


An exploratory analysis was also conducted to examine growth in performance for the 
seven Cohort 1 teachers who submitted complete modules with student work during both the 
2016-2017 and 2017-2018 school years. While significance cannot be determined because of 
the small sample, ratings did improve across time for Dimensions 1 through 3, which focus on 
the foundational aspects of the modules including the effective writing task, standards 
alignment, and fidelity to LDC instruction. 
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6.0 Fidelity of Implementation Analysis 


This chapter describes the results of a broad analysis of the LDC schools’ fidelity of 
implementation to the LDC model. As part of Department of Education’s requirements, UCLA 
CRESST and LDC collaborated to create a fidelity matrix to measure schools’ fidelity of 
implementation across four key components of the LDC model. Within the matrix, each key 
component is measured via multiple indicators, with 14 indicators total across the key 
components. The key components, as depicted in the LDC logic model (see Figure 1.1) and 
described in Chapter 1 are (a) common planning time for LDC professional learning community 
with synchronous coach support; (b) asynchronous support from LDC coaches; (c) teacher 
implementation activities; and (d) leadership support. 


The fidelity matrix can be found in Appendix G. The matrix outlines a process whereby 
fidelity scores on each indicator are computed for each school, and for the program as a whole. 
The matrix also produces a score for whether the program met fidelity for each of the four 
components. Although the process produces school-level fidelity scores for all indicators, the 
initial unit of implementation is either teacher, module, or school depending on the indicator. 
LDC and CRESST staff worked collaboratively to construct the fidelity matrix, with LDC staff 
setting the thresholds for adequate implementation on each indicator and at each level of 
implementation (teacher/module, school, and program). 


Data for these 14 fidelity indicators come from four sources. Three indicators are based 
on data collected via CRESST’s teacher survey. Data on these selected survey items are used to 
produce school- and program-level fidelity scores. Three indicators are based on data from the 
PLC Reflection Form collected by LDC, which was filled out by teacher leaders at the school level 
and used to capture information on both full PLC sessions and one-on-one planning and 
progress calls between the teacher leader and coach. Two indicators are based on LDC 
administrative records capturing attendance by teacher leaders and administrators at in-person 
meetings occurring four times per year. Finally, six indicators are based on LDC’s CoreTools 
analytic data capturing a variety of behaviors including teachers’ viewing of LDC online course 
content in the LEARN portion of CoreTools, editing of modules, and uploading of student work; 
and coaches’ commenting and providing peer review on modules. This report explores and 
presents the overall amount of editing and commenting behavior of teacher participants in 
Chapter 4. Here we focus on whether participants edited key portions of the module, as well as 
the comments provided by coaches on modules as a measure of their asynchronous feedback. 


In this chapter, we will first present the overall results on school- and program-level 
fidelity scores for all the indicators. We will then provide more detailed descriptive results with 
sections for each of the four key components and subsections for each of the 14 fidelity 
metrics. 
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6.1 School- and Program-Level Fidelity of Implementation Scores 


Table 6.1 provides a summary for how LDC schools performed across the four key 
components and 14 indicators of fidelity of implementation. As described in detail in the fidelity 
matrix in Appendix G, schools were rated on 4-point scales (from 0 to 3) on each indicator. At 
the school level, adequate implementation was defined as a score of at least 2. For many of the 
indicators, the 4-point scale is based on the proportion of teachers who met a certain 
implementation threshold. In order for the program as a whole to meet fidelity on a particular 
indicator, a certain proportion of schools had to meet the school-level threshold of 2. For the 
indicators under Key Component 2, program-level fidelity was met if half or more of the schools 
met fidelity. For all of the other indicators (under Key Components 1, 3, and 4) program-level 
fidelity was met if three quarters of schools met fidelity. 


As displayed in Figure 6.1, there was great variation across schools in their success in 
meeting implementation thresholds. Schools met fidelity on a range between zero and 10 
indicators. On average schools met fidelity on just under five indicators. No schools met fidelity 
on more than 10 indicators, suggesting that even high performance schools have room for 
growth. 


Overall, the analysis revealed that a majority of the schools were not able to meet the 
adequate implementation thresholds. Adequate implementation at the program level was met 
on just two of the 14 indicators: Administrator Attendance at Quarterly In-Person Administrator 
Meetings and Teacher Leader Attendance at Quarterly In-Person Teacher Leader Meetings. A 
majority of the schools also met the threshold for Teacher Attendance at Weekly PLC Meetings, 
although that proportion was not sufficient for the program to be labeled as having adequate 
implementation on that indicator. For all the other indicators, a minority of the schools met the 
threshold. For fidelity to be met on a key component, the fidelity matrix requires that fidelity be 
met on each of the indicators for that key component. As a result, the analysis for 2017-2018 
concludes that fidelity was not met for the four key components. 
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Table 6.1 
Summary Table of School- and Program-Level Fidelity Scores by Indicator in 2017-2018 


% of % of % of % of % of 
schools schools schools schools schools Program 
withno  atLevel atLevel atLevel  atLevel met 
Key component Indicator data 0 1 2 3 fidelity? 
Key Component _ Teacher attendance at 2 40 4 16 38 No 
1: Common weekly PLC meetings 
Planning Time for . 
. Amount of time spent on 0 0 62 38 0 No 
LDC Professional ; 
. LDC during common 
Learning : : 
Community With plspmine ie 
Synchronous Exposure to LDC LEARN 2 96 2 0 0 No 
Coach Support content during first 
instructional cycle 
Exposure to LDC LEARN 2 98 0 0 0 No 
content during second 
instructional cycle 
Perceived effectiveness of 4 36 16 22 22 No 
engagement in PLC on 
teacher competencies 
Key Component Coach comments on 11 36 9 16 29 No 
2: Asynchronous modules 
suppert tion Coach formative peer 9 80 2 2 7 No 
LDC Coaches : 
review on modules 
Teacher perception of the 4 53 11 16 16 No 
helpfulness of coach 
written feedback on 
modules 
Key Component Module editing 2 62 9 22 4 No 
3: Teacher : . 
: Module implementation 2 78 7 4 9 No 
Implementation 
Activities 
Key Component Frequency of 0 31 40 20 9 No 
4: Leadership coach/teacher leader 
Support at monthly meetings 
Different Levels _ 
Administrator attendance 0 0 18 13 69 Yes 
at quarterly in-person 
administrator meetings 
Teacher leader attendance 0 0 4 22 73 Yes 
at quarterly in-person 
teacher leader meetings 
Principal mini-task 4 64 9 9 14 No 


observation 
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Frequency Distribution of Schools Reaching Adequate 
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Figure 6.1. Frequency distribution of schools reaching adequate implementation on different numbers 
of fidelity indicators. 


6.2 Key Component 1: Common Planning Time for LDC Professional Learning 
Community With Synchronous Coach Support 


Schools varied greatly in their level of fidelity on the five indicators of Key Component 1, 
as shown in Figure 6.2. Schools were most successful with regard to ensuring a high teacher 
attendance rate at the PLC sessions, with 24 out of 45 (53%) of the schools meeting adequate 
implementation on Teacher Attendance at Weekly PLC Meetings. A substantial minority of the 
schools also met fidelity on Amount of Time Spent on LDC During Common Planning Time (38%) 
and Perceived Effectiveness of Engagement in PLC on Teacher Competencies (44%). Meeting 
fidelity on the indicators of exposure to LDC LEARN content was uniformly challenging for 
schools, with no schools reaching the adequate implementation threshold. 
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Figure 6.2. Number of schools reaching different fidelity levels for each Key Component 1 indicator. 


Teacher attendance at weekly PLC meetings. PLC reflection data show that schools varied 
greatly in terms of the number of times their PLCs met over the course of the 2017-2018 school 
year. The number of sessions ranged from five to 25 times, with PLCs on average meeting 13.6 
times. This is important context for the interpretation of the Teacher Attendance at Weekly PLC 
Meetings fidelity indicator. The indicator measures the proportion of teachers who regularly 
attended sessions but does not incorporate the number of times that the PLC met across the 
year. Success in the two facets of attendance does seem to be related, but there were cases 
where teacher attendance rates were high but the school met a below average number of 
times. And likewise there were cases where schools met a large number of times but individual 
attendance rates were poor. 


Attendance rates across the full population of PLC participants varied a great deal, ranging 
from 0 to 100% and averaging 83%. Not surprisingly, teacher leaders in out-of-classroom 
assignments (n = 9) and teacher leaders who also taught in classrooms (n = 43) attended at 
higher average rates than teachers not playing a leadership role (n = 243): 88% and 92% 
respectively in contrast to 81%. The adequate implementation threshold for individual teacher 
attendance was 80% or greater, and 204 out of 295 PLC participants (69%) reached this 
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threshold. As seen in Figure 6.1, while a majority of teachers in a majority of schools reached 
this threshold, there was a very substantial minority of schools that experienced low 
attendance rates. More specifically, in 40% of schools less than a quarter of the teachers 
reached the adequate implementation threshold of an 80% attendance rate. 


Amount of time spent on LDC during common planning time. This metric is based on PLC 
reflection data for the subset of PLC sessions that coaches joined either digitally or in person. 
Overall, teacher leaders reported that 35 sessions were under 45 minutes long (6%), 329 
sessions were between 45 and 59 minutes (55%), 212 sessions were between 60 and 74 
minutes (36%), and 21 sessions were between 75 and 90 minutes (4%). 


School-level scores were based on the modal (most common) response on the PLC 
reflection form. For 28 of the schools, that response was 45-59 minutes, while for the other 17 
schools, the most common response was 60-74 minutes. Because of the nature of the data, we 
cannot ascertain the exact length of the sessions. As such it is possible that many of the 
sessions recorded as 45-59 minutes came quite close to the desired 60 minutes of PLC time. 
Nevertheless, according to the thresholds set, a majority of the schools did not meet fidelity on 
this indicator. It may make sense for LDC to update its instrument to collect more fine-grained 
data on this indicator. 


Exposure to LDC LEARN content during first and second instructional cycles. Here we 
report on the two indicators measuring PLC participants’ engagement with the LEARN online 
course content (instructional courses that help LDC participants learn how to navigate an LDC 
instructional cycle). Of the 295 teachers and teacher leaders participating in LDC PLCs in 2017-— 
2018, 139 (47%) did not view any LEARN content through their user accounts. The remaining 
156 participants viewed at least one session in one instructional cycle. And a subset of that 
group (n = 71) viewed at least one session in a second instructional cycle. Overall the full 
population of teachers and teacher leaders on average were exposed to 18% of the LEARN 
sessions in the first instructional cycle and 9% of the LEARN sessions in the second instructional 
cycle. 


It is possible that teachers were not complying with instructions to engage with the 
LEARN content within their own user accounts but were still exposed to some of the content in 
the group setting of the PLC. Nevertheless, even if we examine only those participants with any 
exposure to LDC in the first instructional cycle (n = 158), that smaller group still only was 
exposed to an average of 35% of the sessions. Most teachers were not being exposed to the 
majority of LEARN sessions in their first instructional cycle, with even lower results in the 
second instructional cycle. 


Due to the overall low fidelity at the teacher level, none of the schools reached adequate 
implementation on either of the two Exposure to LEARN Content indicators. In fact, only one 
school reached the school-level score of 1 (low) by having 65% of its PLC participants meeting 
the adequate implementation threshold. All other schools scored zero, the lowest fidelity level. 
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The overall low performance of schools on this indicator, however, masks variation across the 
schools in performance. Most significantly, nine out of 45 of the schools (20%) had zero 
teachers with exposure to LEARN content via their user accounts. 


Perceived effectiveness of engagement in PLC on teacher competencies. This indicator is 
based on the 236 teachers who provided data on Question 30 in the teacher survey (see 
Appendix A), which asks teachers to report how much their skills had improved in a number of 
areas aligned with LDC core competencies. Here we base fidelity levels on an index averaging 
the seven-question 30 items, which are each measured on a 1 to 4 scale. The threshold for 
adequate implementation at the teacher level is an index score of 3, corresponding to the 
survey response reporting moderate improvement in the skill area. Across the whole sample, 
15 teachers had scores of 1 to less than 2 (no to a little improvement), 59 had scores of 2 to less 
than 3 (a little to moderate improvement), 131 had scores of 3 to less than 4 (moderate to 
great improvement), and 31 had scores of 4 (great deal of improvement). Just over two thirds 
of respondents in the sample reached the adequate implementation threshold score of 3 on 
the index. 


There was a great deal of variation in survey responses both within and across schools. In 
13 schools, half or fewer of the teachers reported a moderate effect on the competencies. On 
the other hand, 100% of respondents in nine schools reported a moderate or greater effect. 
The variation can be clearly seen in Figure 6.2. Despite many teachers reporting an impact on 
their competencies, a minority of schools had three quarters or more of their teachers 
reporting a moderate impact. 


6.3 Key Component 2: Asynchronous Support From LDC Coaches 


As with Key Component 1, there was a great deal of variation across schools in their level 
of fidelity on the three Key Component 2 indicators, as shown in Figure 6.3. Data were most 
likely to indicate high fidelity on the Coach Comments on Modules indicator, with 20 out of 45 
schools (44%) reaching the adequate implementation threshold. In addition, about a third of 
schools met fidelity on the Teacher Perception of the Helpfulness of Coach Written Feedback on 
Modules indicator. Coach peer review was not a broadly used tool, and as a result only a 
handful of schools met fidelity on this indicator. 
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Figure 6.3. Number of schools reaching different fidelity levels for each Key Component 2 indicator. 


Coach comments on modules. Results on this indicator are based on the universe of 
modules linked to a LEARN instructional cycle. In total there were 162 modules linked to 
courses across 40 of the 45 schools, with the number of linked modules per school ranging from 
one to nine. Those modules in turn received between zero and 13 comments from coaches. An 
adequate level of feedback on a linked module was defined as having received two or more 
coach comments, and 101 out of the total 162 modules (62%) met that threshold. The 
proportion of modules within schools that met the two-comment threshold varied a great deal 
across schools, which accounts for the variation in school-level fidelity scores as seen in Figure 
6.3. 


Coach formative peer review on modules. Results on this indicator are based on the 
universe of teachers and teacher leaders who coauthored at least one linked course. That 
population includes 221 participants across 41 schools. At the teacher level, fidelity was defined 
as having received at least one peer review from a coach. Only 34 teachers received at least 
one peer review from their coach, and therefore less than 10% of the schools met the school- 
level fidelity threshold. 


Teacher perception of helpfulness of coach written feedback on modules. Across 246 
teacher survey respondents, 141 teachers (57%) reported using written feedback on modules 
and finding it moderately to very helpful. Across schools, the proportion of teachers who 
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reported at least moderate usefulness ranged from zero to 100% with a mean of 59% of 
teachers. Most schools did not meet the school-level threshold for the perceived helpfulness of 
written feedback. 


6.4 Key Component 3: Teacher Implementation Activities 


Here we report on indicators related to teachers’ engagement with the process of 
designing and implementing LDC modules. As shown in Figure 6.4, schools were more 
successful in terms of meeting thresholds for the proportion of their teachers who engaged 
deeply in the design process by editing key portions of modules, with just over a quarter of 
schools meeting fidelity on this indicator. In only about one eighth of schools did three quarters 
or more of teachers implement two modules, as measured by the uploading of student work 
samples. 
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Figure 6.4. Number of schools reaching different fidelity levels for each Key Component 3 indicator. 


Module editing. As outlined in the fidelity matrix, teacher-level fidelity levels are built in a 
stepladder fashion. To meet Fidelity Level 1, a teacher needed to edit at least one task in a 
module. To meet Fidelity Level 2 (adequate implementation), a teacher needed to reach the 
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Level 1 threshold and edit either standards or text in at least one module. To reach Fidelity 
Level 3, a teacher had to meet the previous requirements and also edit either skills and mini- 
tasks or the rubric. 


A total of 150 participants (51%) failed to meet the threshold of editing the teaching task 
in one module and therefore implemented at a fidelity level of zero. Nine participants (3%) 
edited a task, but did not edit standards or texts, and therefore were at Fidelity Level 1. A total 
of 136 participants (46%) met the adequate implementation threshold, with 38 (13%) scoring at 
Level 2 (edited standards or texts but not skills/mini-tasks or rubric) and 98 (33%) scoring at 
Level 3 (edited standards or text AND skills/mini-tasks or rubric). As a result of less than half of 
participants overall meeting the teacher-level adequate implementation threshold, nearly three 
quarters of schools did not meet the school level threshold for adequate implementation. 


Module implementation. For this indicator, our sample is the 283 classroom teachers 
who participated in LDC in 2017-2018 (as the metric involves classroom implementation, out- 
of-classroom staff are excluded). The upload of student work serves as a proxy for whether the 
teacher implemented a module in her classroom. The number of modules with uploaded 
student work ranged from zero to nine with a mean of just over one. One hundred teachers 
(36%) didn’t upload student work to any modules. Eighty-nine (32%) uploaded student work to 
one module, and 73 (26%) uploaded to two modules. Twenty teachers (7%) uploaded student 
work to three or more modules. Overall, just one third of teachers met the adequate 
implementation threshold, and as a result only 13% of schools met fidelity on the indicator. 


6.5 Key Component 4: Leadership Support at Different Levels 


As depicted in Figure 6.5, there was a high level of fidelity with regard to attendance of 
administrators and teacher leaders at quarterly in-person meetings. Results, however, were 
considerably less positive for the other two indicators of leadership support. Most teacher 
leaders did not report meeting with coaches at the recommended frequency. And most 
teachers reported that they were not observed by principals implementing LDC instruction. 
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Figure 6.5. Number of schools reaching different fidelity levels for each Key Component 3 indicator. 


Frequency of coach/teacher leader monthly meetings. Teacher leaders reported meeting 
with their coach between zero and 25 times with a mean of 6.6 times, according to PLC 
reflection data. The adequate implementation threshold was set at nine or more coach/teacher 
leader meetings across the school year, and only 13 of 45 schools (29%) met that threshold. 
Eighteen of 45 schools (40%) met between four and eight times (low implementation) and 
nearly a third met less than four times (very low implementation). 


Administrator attendance at quarterly in-person administrator meetings. Administrators 
had the opportunity to attend four quarterly events. Administrators in 13 schools (29%) 
attended all four events, in 18 schools (40%) attended three events, in six schools (13%) 
attended two events, and in eight schools (18%) attended just one event. With the adequate 
implementation threshold set at two of four events, 37 out of 45 schools (82%) met the 
threshold. 


Teacher leader attendance at quarterly in-person teacher leader meetings. Teacher 
leaders also had the opportunity to attend 4 quarterly events. Teacher leaders attended all four 
events in 20 schools (44%), attended three events in 13 schools (29%), attended two events in 
10 schools (22%), and attended just one event in two schools (4%). With the adequate 
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implementation threshold again set at two of four events, 43 out of 45 schools (96%) met the 
threshold. 


Principal mini-task observation. Fifty percent of teacher survey respondents reported 
never having been observed by an administrator when teaching an LDC mini-task, 23% reported 
being observed once, 13% reported being observed twice, and 11% reported being observed 
three or more times. With only half of all respondents reported having been observed, over 
two thirds of schools did not meet the adequate implementation threshold of three quarters of 
teachers having been observed. 


6.6 Summary of Results 


In summary, the fidelity matrix analysis revealed that none of the schools met fidelity 
requirements on the four key components in 2017-2018 in the West Coast district. The ability 
of PLCs to set aside common planning time that worked for all teachers varied across schools. 
The frequency of meetings ranged broadly, as did the attendance rates of the participants. 
While a majority of schools were able to maintain high attendance rates, a substantial minority 
did not meet the desired attendance threshold. When PLCs did meet, they most often met for 
between 45 and 59 minutes, although many sessions also lasted the desired 60 minutes or 
more. LDC intended for teachers to be exposed to key online course content within PLC time, 
but by and large this did not happen. Many participants did not view any LEARN content in their 
user accounts (although they may have been exposed to the content via a peer or coach’s 
account), and those that did tended to view a minority of sessions in the instructional cycle. The 
evidence suggests that many participants did not embark on a second instructional cycle. Given 
the wide variety of experiences that schools and teachers had with common planning time, it is 
not surprising that teachers reported a variety of attitudes with regard to the effectiveness of 
PLC in building teacher skills. 


The results also reveal gaps related to asynchronous support form coaches. Coaches 
seemed to use the comments function to a great degree more than the peer review function. 
Nevertheless, expectations with regard to coach comments on modules linked to courses were 
not met. Many teachers did not find feedback helpful, and the results on other indicators 
suggest this result may be due in part to the lack of a sufficient amount of feedback. 


Results on module editing suggest that many teachers were not heavily engaged in the 
design process, with half of teachers not having edited any teaching tasks. Those that did edit 
the teaching task tended to also edit either standards or texts, and a third of teachers also 
reached the stage of editing skills/mini-tasks and the rubric. On a positive note, there was 
evidence that nearly two thirds of teacher participants implemented modules in the classroom, 
but just about half of that group met the adequate implementation threshold of uploading 
student work for two modules. 


Leader attendance was a bright spot, with most administrators and teacher leaders 
attending at least two of four meetings. There were gaps, however, in more regular and 
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intensive leadership support. In most schools, teacher leaders did not manage to meet with 
their coaches monthly. And according to teachers, administrators only made it to about half of 
teachers’ classrooms to observe LDC instruction. 


Variation across schools was substantial, with some schools meeting fidelity on as many 
as eight to 10 indicators, and on the other end, some schools struggling with nearly every 
aspect of implementation. No patterns emerged for cohort and school-level subgroups with 
Cohort 1 and Cohort 2 schools and elementary and secondary schools meeting similar levels of 
fidelity, despite great variation within each of the subgroups. Analysis of implementation in 
2018-2019 will yield additional information on how fidelity changes over time. 
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7.0 Student Outcome Analysis 


This section presents the student outcome analysis we conducted to evaluate LDC’s 
impact on student learning in the 2017-2018 school year. As described earlier, our LDC teacher 
sample included both elementary and middle school teachers from two cohorts of LDC schools 
in the study district. Some of the LDC teachers in Cohort 1 schools were in their second year of 
LDC implementation (having participated in 2016-2017) and some of the teachers newly joined 
the existing PLCs in 2017-2018. LDC teachers from Cohort 2 schools were all in their first year 
of participating in LDC in 2017-2018. Separate sampling and analyses were conducted for three 
groups of teachers: Cohort 1 returning middle school teachers, Cohort 2 elementary school 
teachers, and Cohort 2 middle school teachers. 


We could not conduct analyses for the other groups of teachers because of sample sizes. 
More specifically, we could not conduct meaningful analyses of the impact of Cohort 1 
returning or Cohort 1 new elementary school teachers due to the small number of students 
who were taught by these Cohort 1 teachers and for whom we have 2015-2016 baseline data. 
There was also too small a number of students connected to Cohort 1 new middle school 
teachers in 2017-2018 to conduct meaningful analyses. 


We begin this chapter by describing the process we used to define the LDC student 
samples for each analysis and to construct matched comparison samples. We then present 
descriptive statistics for the treatment and comparison groups. Finally, we report the estimated 
impact of LDC on students as measured by the Smarter Balanced Assessment scores in English 
language arts (ELA). 


7.1 LDC Sample and the Matching Process 


As described earlier, our LDC teacher sample included both elementary and middle school 
teachers in the study district. Separate sampling and analyses were conducted for these two 
groups of teachers, as described in Section 2.5 of this report (Analytical Approaches). 


Cohort 1 returning middle school sample. The eligible LDC sample for the Cohort 1 
returning middle school analysis includes all students (a) who were enrolled in one school 
campus for the entire 2017-2018 school year under the instruction of at least one of the 
participating LDC teachers in their second year of LDC participation, and (b) for whom baseline 
achievement scores, outcome year achievement scores, and demographic data were available. 
Achievement and demographic data were used in the matching process. 


The 2017-2018 Cohort 1 returning middle school sample prior to the CEM process 
included four schools, 22 Cohort 1 returning middle school teachers who consented to 
participate in the evaluation study, and their 1,111 students. After the student-level matching, 
our final Cohort 1 returning middle school LDC sample included 995 students and the same 
number of teachers and schools prior to matching (see Table 7.1). 
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Prior to matching, the potential middle school comparison sample consisted of 116 
schools, 3,874 teachers, and 50,664 students. This comparison sample was substantially 
reduced during the first stage of matching, which identified up to five schools that closely 
resembled each of the four LDC middle schools. After student-level CEM, a workable analytic 
comparison sample consisted of 19 schools, 313 teachers, and 995 students. 


Table 7.1 
Before and After Matching Sample Sizes: Cohort 1 Middle School Analysis 


LDC sample Comparison sample 
Stage Schools Teachers Students Schools Teachers Students 
Stage 1 
Before matching 4 22 1,111 116 3,874 50,664 
After matching 4 22 1,111 19 555 11,593 
Stage 2 
After matching 4 22 995 19 313 995 


Cohort 2 elementary sample. As with the Cohort 1 analysis, the eligible LDC sample for 
the Cohort 2 elementary school analysis includes all students (a) who were enrolled in one 
school campus for the entire 2017-2018 school year under the instruction of at least one of the 
participating LDC teachers, and (b) for whom baseline achievement scores, outcome year 
achievement scores, and demographic data were available. Achievement and demographic 
data were used in the two-stage matching process. 


Table 7.2 
Before and After Matching Sample Sizes: Cohort 2 Elementary School Analysis 


LDC sample Comparison sample 
Stage School Teacher Student School Teacher Student 
Stage 1 
Before matching 21 85 1,524 470 3,529 75,898 
After matching 21 85 1,524 103 737 12,168 
Stage 2 
After matching 21 85 1,379 102 509 1,379 
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As reported in Table 7.2, the resulting LDC sample included 21 schools, 85 teachers, and 
1,524 students prior to the CEM process. After the CEM student-level matching, our final 
Cohort 2 elementary LDC sample was reduced to 1,379 students. Prior to matching, the 
potential comparison sample consisted of 470 schools, 3,529 teachers, and 75,898 students. 
This comparison sample was substantially reduced during the first stage of matching, which 
identified up to five schools that most closely resembled each of the 21 LDC schools. After 
student-level CEM, a workable analytic comparison sample of 102 schools, 509 teachers, and 
1,379 students was constructed. 


Cohort 2 middle school sample. The eligible LDC sample includes all students (a) who 
were enrolled in one school campus for the entire 2017-2018 school year under the instruction 
of at least one of the participating LDC teachers, and (b) for whom baseline achievement 
scores, outcome year achievement scores, and demographic data were available. Achievement 
and demographic data were used in the matching process. Again, we employed a two-stage 
matching process to select schools and students. 


As shown in Table 7.3, the resulting Cohort 2 LDC middle school sample included 10 
schools, 43 teachers, and 2,574 students prior to the CEM process. After the CEM student-level 
matching, our final secondary LDC sample was reduced to 2,382 students. 


Table 7.3 
Before and After Matching Sample Sizes: Cohort 2 Middle School Analysis 


LDC sample Comparison sample 
Schools Teachers Students Schools Teachers Students 
Stage 1 
Before matching 10 43 2,574 116 3,876 52,393 
After matching 10 43 2,574 49 1,356 31,109 
Stage 2 
After matching 10 43 2,382 48 807 2,382 


Prior to matching, the potential comparison sample consisted of 116 schools, 3,876 
teachers and 52,393 students. This comparison sample was substantially reduced during the 
first stage of matching, which identified up to five schools that closely resembled each of the 10 
LDC schools. After student-level CEM, a workable analytic comparison sample of 48 schools, 
807 teachers, and 2,382 students was constructed (see Table 7.3). 
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7.2 Descriptive Results on the Matched Analytic Samples 


Tables 7.4, 7.5, and 7.6 present the characteristics of the LDC student and comparison 
students in the final analytical samples for the Cohort 1 returning middle school, Cohort 2 
elementary, and Cohort 2 middle school analyses respectively. Treatment and comparison 
samples matched very closely. Exact matching was achieved on some variables, and all 
demographic variables were within five percentage points. We used the spring 2018 Smarter 
Balanced ELA tests as our outcome measures in all analyses. The Cohort 1 analysis used the 
spring 2016 Smarter Balanced ELA tests as one of the matching variables while the Cohort 2 
analyses used the spring 2017 Smarter Balanced ELA tests. 


For the baseline achievement matching variable, we standardized Smarter Balanced scale 
scores at each grade level relative to district performance, based on the district mean and 
standard deviation for the ELA test at each grade level. Standardizing scores in this way enables 
us to easily compare our samples’ performance relative to the district’s and to compare scores 
across grades and years more easily. A standardized scale score of zero, for example, indicates 
that the student scored at the mean for all other students in the district who took the same 
test. A standardized scale score of 1.0 meant that the student scored one standard deviation 
higher than the district mean. Conversely, a standardized scale score of -1.0 indicated that the 
student scored one standard deviation lower than the district mean. 


The final LDC Cohort 1 middle school student sample after matching was composed 
almost entirely of Hispanic students eligible for free or reduced-price lunches (see Table 7.4). 
The sample was also mostly comprised of students who were in fifth and sixth grade (93.5% 
combined) in the baseline year (and therefore in seventh and eighth grade in 2017-2018). 
English language learners represented over one quarter of this sample, while special education 
students and those classified as gifted represented slightly more than 10% each. Mean 
performance on the baseline year assessments was more than one quarter of a standard 
deviation lower than the districtwide performance levels in both mathematics and ELA. 


The final Cohort 2 LDC elementary student sample (see Table 7.5) was about three 
quarters Hispanic and almost two thirds were eligible for free and reduced-priced lunches. The 
sample was mostly composed of students in third grade (44.8%) and fourth grade (48.7%) in the 
baseline year. English language learners represented slightly less than one quarter of this 
sample, while special education students and those classified as gifted each represented slightly 
less than 10% of the sample. In addition, mean performance on the baseline year assessments 
was slightly higher or about the same for LDC students as compared to districtwide 
performance levels in mathematics and ELA. 


As shown in Table 7.6 the final Cohort 2 LDC middle school student sample included a very 
large proportion of Hispanic students (91.4%), and a majority of students with low 
socioeconomic backgrounds (59.5%). The sample was mostly composed of students that were 
in fifth grade (58.3%) and sixth grade (35.2%) in the baseline year. English language learners 
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represented slightly more than one tenth of this sample, while there were about twice as many 
gifted (16.8%) than special education students (8.1%). In addition, mean performance on the 
baseline year assessment was about the same for LDC students as compared to districtwide 
performance levels in mathematics and ELA. 


Table 7.4 


2015-2016 Student Characteristics of the Cohort 1 Returning Middle School Treatment 
and Comparison Groups After Matching 


Treatment group Comparison group 
Student characteristic (n=995) (n =995) 


Race/ethnicity 


Hispanic (%) 97.6 97.6 
Black (%) 1.7 1.7 
Asian (%) 0.2 0.2 
White (%) 0.3 0.3 
Other (%) 0.2 0.2 
Female (%) 50.1 50.1 


Special programs status 


Poverty (%) 95.6 93.1 
English language learner (%) 29.3 29.3 
Special education (%) 11.8 14.2 
Gifted (%) 11.4 12.6 


Student baseline achievement 
Mean baseline year mathematics Z score -0.254 -0.221 
Mean baseline year ELA Z score -0.299 -0.302 
Class and teacher characteristics 
Mean baseline ELA Z score of current peers -0.306 -0.260 
Teacher years of experience 96.8 95.7 


Grade level at baseline year 


Grade 4 in 2015-2016 (%) 6.5 6.5 
Grade 5 in 2015-2016 (%) 58.3 58.3 
Grade 6 in 2015-2016 (%) 35.2 35.2 
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Table 7.5 
2016-2017 Student Characteristics of the Cohort 2 Elementary School Treatment and 
Comparison Groups After Matching 


Treatment group Comparison group 
Student characteristic (n = 1,379) (n = 1,379) 


Race/ethnicity 


Hispanic (%) 74.0 74.0 
Black (%) 10.2 10.2 
Asian (%) 3.7 3.9 
White (%) 9.1 10.0 
Other (%) 3.0 1.9 
Female (%) 51.0 51.0 


Special programs status 


Poverty (%) 62.7 62.4 

English language learner (%) 21.4 21.4 

Special education (%) 7.8 10.6 

Gifted (%) 9.7 8.3 
Student baseline achievement 

Mean baseline year mathematics Z score 0.087 0.057 

Mean baseline year ELA Z score 0.003 0.019 


Class and teacher characteristics 
Mean baseline ELA Z score of current peers -0.026 -0.000 
Teacher years of experience 16.9 13.4 


Grade level at baseline year 


Grade 3 in 2016-2017 (%) 44.8 44.8 
Grade 4 in 2016-2017 (%) 48.7 48.7 
Grade 5 in 2016-2017 (%) 6.5 6.5 
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Table 7.6 


2016-2017 Student Characteristics of the Cohort 2 Middle School Treatment and Comparison Groups 


After Matching 


Treatment group Comparison group 


Student characteristic (n = 2,382) (n = 2,382) 

Race/ethnicity 

Hispanic (%) 91.4 91.4 

Black (%) 3.6 3.6 

Asian (%) 1.1 1.3 

White (%) 2.6 3.0 

Other (%) 1.3 0.7 
Female (%) 49.3 49.3 
Special programs status 

Poverty (%) 59.5 60.4 

English language learner (%) 11.4 11.4 

Special education (%) 8.1 8.1 

Gifted (%) 16.8 18.6 
Student baseline achievement 

Mean baseline year mathematics Z score -0.010 0.012 

Mean baseline year ELA Z score -0.011 0.003 
Class & teacher characteristics 

Mean baseline ELA Z score of current peers -0.028 -0.058 

Teacher years of experience 14.8 10.5 
Grade level 

Grade 5 in 2016-2017 (%) 11.7 11.7 

Grade 6 in 2016-2017 (%) 46.2 46.2 

Grade 7 in 2016-2017 (%) 42.1 42.1 


7.3 Outcome Analysis Results: Cohort 1 Middle School Sample 


As discussed in detail in Chapter 2, middle school students could be exposed to between 
zero and six different LDC teachers in a given year. Using an MMMC design, each observation at 
Level 1 represented one student, who was linked to the ELA, social studies/history and science 
teachers that student was exposed to during the year. Weights across teachers for each student 


summed to a unity (1). 
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Two different approaches were used to model the LDC treatment intervention variable as 
a fixed effect at the student level. The first model was dosage dependent, and took into 
account variation in middle school students’ level of exposure to LDC teachers. In this approach, 
the treatment variable was structured as a continuous response variable between zero and 
one, based on exposure to LDC teachers for ELA, social studies/history and/or science. In 
contrast, the second approach was modeled as dosage independent, and considered any 
student exposed to any LDC intervention teacher in at least one subject to be a treated 
individual. In this latter approach the treatment variable was dichotomous, coded as one for 
LDC treated students and zero for comparison students. 


Table 7.7 


Cohort 1 Returning LDC Middle School Teacher Effect Estimates on 2017-2018 Smarter Balanced ELA 
Performance, Dosage-Dependent and Dosage-Independent Models 


Dosage-dependent model Dosage-independent model 
Variables coefficient (SE) coefficient (SF) 
Level 2 LDC teacher treatment 0.118 (0.116) 0.050 (0.083) 


Level 1 student characteristics 
Hispanic 
Black 
Poverty 
Female 
English language learner 
Special education 
Gifted 
Honors English 


Baseline peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


-0.216 (0.127) 
-0.271 (0.151) 
-0.077 (0.046) 
0.173 (0.022)* 
-0.113 (0.029)* 
0.075 (0.040) 
0.018 (0.040) 
0.169 (0.056)* 
0.109 (0.057) 
0.189 (0.020)* 
0.439 (0.021)* 


-0.215 (0.127) 
-0.270 (0.151) 
-0.078 (0.046) 
0.173 (0.022)* 
-0.113 (0.029)* 
0.075 (0.040) 
0.018 (0.040) 
0.169 (0.056)* 
0.109 (0.057) 
0.189 (0.020)* 
0.439 (0.021)* 


Note. Based on the dosage-dependent model, the average treated student received a 0.485 treatment dosage. 
Because of this, using the dosage-dependent model we could estimate an average treatment on the treated (ATT) 
at (0.485 x 0.118) = 0.057. 

*o = .05. 


In Table 7.7 we present results of both the dosage-dependent and dosage-independent 
models on middle school students’ ELA performance in 2017-2018. As can be seen, model 
results for the LDC effect on student outcomes are in the positive direction, but are not 
statistically significant for either model. In other words, neither analysis provided sufficient 
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evidence to conclude that students taught by LDC teachers performed better on the ELA test 
than did their matched peers in the comparison group. 


The significant effects of the covariates on student performance also were similar under 
the two models and were in the expected directions. Baseline ELA performance was the 
strongest predictor and baseline mathematics performance also helped explain the outcome. In 
addition to baseline achievement, three demographic variables helped predict performance: 
English language learners performed at lower levels than English only and reclassified fluent 
English proficient students, females performed at significantly higher levels than males, and 
students enrolled in honors English courses performed at higher levels than did their peers 
taking standard English courses. 


7.4 Outcome Analysis Results: Cohort 2 Elementary Sample 


In general, elementary students who were enrolled in the same school for the entire year 
were connected to just one teacher. On rare occasions, however, elementary students were 
taught by multiple teachers rather than a single one. For this reason, and to allow all teachers 
into the analysis we once again employed an MMMC design. The outcome was students’ 
Smarter Balanced scores in ELA. For technical reasons related to evaluating effect sizes, Smarter 
Balanced scale scores were standardized to the study sample. 


As discussed in detail in Chapter 2, students could be exposed to between zero and three 
different LDC teachers at the elementary school level. Using an MMMC design, each 
observation at Level 1 represented one student, who was linked to the ELA, social 
studies/history, and science teachers that a student was exposed to during the year. Weights 
across teachers for each student summed to a unity (1). 


In the middle school analysis, we presented results of models that are both dosage 
dependent and dosage independent. Since the great majority (99%) of students in the Cohort 2 
elementary analysis were connected with a single teacher, findings for the two models become 
nearly identical. As a result, we only present the dosage-independent model. As shown in Table 
6.8, model results show no statistically discernible LDC effect on the student outcome. In other 
words, students taught by LDC teachers scored similarly on the ELA test to their matched peers 
in the comparison group. 


The effects of the covariates on student performance were similar to those in the Cohort 
1 middle school model, but included some additional significant variables. In addition to 
baseline ELA and mathematics performance, English language learner status, and gender, the 
variables of Black ethnicity, Hispanic ethnicity, special education, and average baseline ELA 
achievement of a student’s peers were all significant predictors of ELA performance and were 
in the expected directions. 
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Table 7.8 


Cohort 2 LDC Elementary School Teacher Effect Estimates on 2017- 
2018 Smarter Balanced ELA Performance, Dosage-Independent Model 


Variables Dosage-independent 
model coefficient (SE) 


Level 1 LDC teacher treatment 0.026 (0.034) 


Level 2 student characteristics 


Hispanic -0.058 (0.028) * 
Black -0.166 (0.040) * 
Poverty -0.029 (0.019) 
Female 0.100 (0.017) * 
English language learner -0.110 (0.030) * 
Special education -0.093 (0.031) * 
Gifted 0.014 (0.034) 
% teachers with < 3 years of experience -0.001 (0.000) 
Baseline peer ELA Z score 0.056 (0.026) * 
Baseline year mathematics Z score 0.223 (0.017) * 
Baseline year ELA Z score 0.501 (0.017) * 
*9 = .05. 


7.4 Outcome Analysis Results: Cohort 2 Middle School Sample 


Similar analyses to those conducted for Cohort 1 middle schools were conducted for the 
Cohort 2 middle school sample. At the middle school level, the norm was student exposure to 
multiple teachers. As discussed in detail in Chapter 2, students could be exposed to anywhere 
between zero and six different LDC teachers. Using an MMMC design, each observation at Level 
1 represented one student, which was linked to the ELA, social studies/history, and science 
teachers that student was exposed to during the year. Weights across teachers for each student 
summed to a unity (1). 


In Table 7.9 we present results of models that are both dosage dependent and dosage 
independent. Dosage-dependent model results indicate a statistically significant and positive 
LDC effect on the student outcome. In contrast, the dosage-independent model did not yield a 
statistically significant LDC effect. The significant dosage-dependent effect suggests that 
increased student exposure to LDC teachers was connected to improved ELA test performance. 
More specifically, treatment students with exposure to LDC in all three subjects were estimated 
to perform 0.149 standard deviations above matched comparison students. The average 
treatment student, however, had a 44% exposure to LDC in core content classes, and is 
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estimated to perform 0.066 standard deviations above matched comparison students (the 


average treatment on treated effect). 


Table 7.9 


Cohort 2 LDC Middle School Teacher Effect Estimates on 2017-2018 Smarter Balanced ELA 


Performance 


Variables 


Dosage-dependent 
model coefficient (SE) 


Dosage-independent 
model coefficient (SE) 


Level 1 LDC teacher treatment 
Level 2 student characteristics 

Hispanic 

Black 

Poverty 

Female 

English language learner 

Special education 

Gifted 

Honors course 

% teachers with < 3 years of experience 

Baseline peer ELA Z score 

Baseline year mathematics Z score 


Baseline year ELA Z score 


0.149 (0.063) * 


-0.042 (0.034) 
-0.178 (0.050) * 
-0.019 (0.014) 
0.104 (0.013) * 
-0.113 (0.025) * 
-0.108 (0.028) * 
0.059 (0.021) * 
0.146 (0.028) * 
-0.000 (0.001) 
0.003 (0.028) 
0.220 (0.013) 
0.513 (0.013) * 


0.055 (0.038) 


-0.042 (0.034) 
-0.180 (0.050) * 
-0.019 (0.014) 
0.104 (0.013) * 
-0.112 (0.025) * 
-0.108 (0.028) * 
0.059 (0.021) * 
0.148 (0.028) * 
-0.000 (0.001) 
0.002 (0.028) 
0.221 (0.013) * 
0.513 (0.013) * 


Note. For the dosage-dependent model, since the average treatment student received a 0.440 treatment 
dosage, we could estimate an average treatment on the treated (ATET) at 0.44 x 0.149 = 0.066. 
KH 

p=.05. 


7.5 Summary and Interpretation of Results 


Here we summarize the quasi-experimental results and provide a lens through which the 
reader can contextualize the magnitude of the results. In Figures 7.1 and 7.2, we present the 
dosage-dependent effects for each of the three analyses. Figure 7.1 depicts the estimated 
impacts of LDC in the three samples on students exposed to LDC teachers in all three major 
content areas: ELA, social studies/history, and science. The effect sizes for these estimates can 
be best understood as the estimated impact of LDC under ideal conditions. Figure 7.2 depicts 
the estimated impact of LDC in the three samples on the average observed student, who in the 
middle school context had considerably less exposure to LDC teachers in her core content 
classes. 
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Teacher Cohorts 


ES Cohort 1 Middle School Teachers (n=22) 
HE Cohort 2 Elementary Teachers (n=85) 
ES Cohort 2 Middle School Teachers (n=43) 


Figure 7.1. Treatment effect on 2017-2018 Smarter Balanced ELA scores with 95% confidence interval 
for students with full LDC dosage, by cohort. 


Teacher Cohorts 
| Cohort 1 Middle School Teachers (n=22) 


S Cohort 2 Elementary Teachers (n=85) 
ES cohort 2 Middle School Teachers (n=43) 


Figure 7.2. Treatment effect on 2017-2018 Smarter Balanced ELA scores with 95% confidence interval 
for students with average LDC dosage, by cohort. 


As can be seen in the figures, the lower bound of the confidence intervals around the 
estimates for the impact of Cohort 2 middle school teachers is above zero. For the other two 
analyses, the confidence intervals cross the zero line, and therefore the estimates are not 
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statistically significant at the 95% level. Not that the estimates for the impact of Cohort 1 
middle school teachers are similar to those of the Cohort 2 middle school teachers. The 
confidence intervals, however, are much wider due to a considerably smaller sample size of 
teachers and associated students. Attrition was certainly a factor in the precision of the 
estimate for Cohort 1 teachers. 


The vast majority of elementary students were associated with just one teacher who 
taught them in all three core content areas. As a result, dosage for elementary school students 
was nearly identical in the two models, and the effect estimates produced were nearly identical 
as well. 


To help the reader contextualize the statistically significant effects for Cohort 2 middle 
school teachers, we utilize an approach developed by Hill et al. (2008), which involves 
benchmarking against average student gains over the course of a school year. The authors 
reviewed annual achievement gains in seven nationally normed reading assessments: California 
Achievement Test (CST) fifth edition, Stanford Achievement Test (SAT) ninth edition, 
TerraNova-Comprehensive Test of Basic Skills (CTBS), Gates-MacGinitie Reading Test, 
Metropolitan Achievement test (MAT) eight edition, Terra Nova-CAT, and SAT tenth edition. 
They found that students gained an average of .32 standard deviations from Grade 5 to 6, .23 
standard deviations from Grade 6 to 7, and .26 standard deviations from Grade 7 to 8. A simple 
mean of these three average gains is .27. 


Using this benchmark, and assuming a 9-month school year, the .066 effect estimate for 
students with average observed LDC dosage is of a similar magnitude to 2.2 months of learning 
in the Hill et al. (2008) meta-analysis [(.066/.27)*9=2.2]. Likewise, the .149 effect estimate for 
students with full LDC dosage aligns to approximately 5 months of schooling [.149/.27)*9=5]. It 
is important to note again that the ideal conditions of students being exposed to LDC in all 
three core content areas across the whole school year was not met for most students; 
therefore, the extrapolation of 2.2 months is the figure best aligned with the actual observed 
effect of LDC. 


While the results are encouraging, it is important to note that results in this report are 
exploratory. The statistically significant effects found were for teachers in their first year of 
implementation, while the evaluation’s confirmatory research questions focus solely on 
teachers in their second year of implementation. Confirmatory results will depend partially on 
Smarter Balanced outcome scores in spring of 2019, and the ability of QED analyses to detect 
effects will depend on sample size (and therefore attrition). Another reason for caution is that 
the study by HIll et al. (2008) is over 10 years old, and Smarter Balanced may have different 
average gains. Nevertheless, the extrapolations offer a useful way to benchmark effect sizes 
against expected learning in the middle school years, and help contextualize the magnitude of 
the results. 
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8.0 Summary of Findings 


This annual report examines LDC following two years of implementation in Cohort 1 
schools and one year of implementation in Cohort 2 schools. We summarize these results 
organized by the three categories of evaluation questions we listed in Chapter 1 of this report. 


8.1 Program Characteristics and Implementation 


Consistent with prior year results, participants reported overwhelmingly positive attitudes 
toward LDC and its implementation at their schools. Teachers generally appreciated the 
opportunity to collaborate with colleagues, and nearly uniformly praised their LDC coaches. 
Prior to 2017-2018, LDC made further refinements to the content, sequencing, and delivery of 
CoreTools’ online courses with the intent of streamlining the learning process for PLC members. 
To some extent this effort seems to have been successful as open-ended responses in our 
survey did not reference instructional content as a problem (as was common in the pilot year). 
Analysis of CoreTools data, however, revealed that teachers are being exposed to a relatively 
small proportion of the content that LDC intended for them to engage with. 


In 2017-2018, local PLC leadership was assigned to a teacher leader rather than a project 
liaison. The role was strengthened by providing a stipend to the individual, and principals and 
assistant principals were no longer permitted to play that role. Again, these program changes 
seem to have been effective. PLC members overwhelmingly reported that their teacher leaders 
were supportive, knowledgeable, and helpful. Teacher leaders themselves also reported high 
satisfaction with support from coaches, professional development offerings, and how the 
teacher leader role allowed them to be instructional leaders in their schools. 


Evidence suggest that implementation was largely faithful to the intended structure of 
LDC, although the level of implementation was not uniformly high. Most PLCs met at least every 
other week as they were expected to do, but some PLCs struggled to meet frequently and 
ensure high attendance. Coaches provided feedback through multiple mechanisms, but PLC 
members did not always find the feedback useful. Analysis of program data suggests that nearly 
all participants were engaging with the module building platform, although that engagement 
did vary greatly across individuals. Module analysis suggests that the materials adapted and 
created by PLC members varied in levels of completion, however, and based on presence of 
uploaded student work, a sizable number of modules might not have been implemented in the 
classroom. 


An encouraging development was the increase in average quality of modules relative to 
prior years. This finding is somewhat tempered by the fact that our exploratory analysis did not 
show an increase in quality from 2016-2017 to 2017-2018 for Cohort 1 teachers who 
completed modules in both years. The sample size for that growth analysis was, however, very 
small, and we anticipate that with lower attrition in Cohort 2, we will be able to learn more 
about change in module quality within teachers over time. 
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8.2 Contextual Factors and Implementation 


As previously noted, coaches and teacher leaders were almost universally praised by 
teachers participating in LDC. Teachers also generally reported that their school administrators 
were supportive of the program, although administrators’ level of participation in PLC meetings 
and observation of LDC instruction varied greatly across the sample. Survey respondents were 
less confident that district-level administrators were knowledgeable about and supportive of 
the LDC program in their schools. 


As in prior years, adequate time was a frequently cited concern, with some respondents 
hoping for more collaborative time, more prep time, time before the start of the year, and/or 
compensated time. Technology also remained a problem for some participants. Some 
respondents were frustrated with technical problems that marred the experience of Zoom 
meetings, and some respondents simply would have preferred in-person rather than digital 
meetings with coaches. On a positive note, respondents seemed to report less difficulty with 
the CoreTools platform than in prior years, which may relate to both increased skills on the part 
of Cohort 1 participants, and refinements made by LDC to the platform. 


One new issue that surfaced this year was the challenge faced by new teacher 
participants joining established PLCs. Survey responses from this subgroup were less 
enthusiastic in a number of areas, such as effectiveness of the PLC, usefulness of coach 
feedback, and their own capacity to be a part of shaping the initiative. CoreTools analyses 
showed that Cohort 1 new teachers demonstrated less engagement in terms of viewing, 
editing, and commenting on modules. These findings suggest that new participants may have 
felt that the program was not sufficiently differentiated for their needs. 


8.3 Program Impacts 


Based on survey results, LDC was perceived to have positive impacts on a range of both 
teacher practices and student skills. Teachers in general felt that LDC had improved their 
instructional planning and pedagogical skill sets and was also helping to promote collaboration 
between teachers. Module analysis suggests that overall mean module quality was higher in 
2017-2018 than 2016-2017. 


Sufficient sample size was available to conduct quasi-experimental tests of the impact of 
LDC on students under Cohort 1 returning middle school teachers, and under Cohort 2 
elementary and middle school students. All effect estimates were in the positive direction, with 
a Statistically significant effect found for Cohort 2 middle school teachers when LDC treatment 
was treated as a continuous variable accounting for students’ different levels of exposure to 
LDC teachers. 
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Appendix A: 
Teacher Survey and Responses 


2017-2018 
LDC Teacher Survey 


LDC Participation 


T1. Please select your school from the drop down box. 
Teachers are skipped to T3 if they teach in an elementary school. 


T2a. Inthe current school year (2017-18), how many classes did you teach? 


| | classes n=91, Mean = 4.62, Range: 1-7 


T2b. In how many of these classes did you use LDC modules and/or mini-tasks? 


| | classes n=91, Mean = 2.70, Range: 0-7 


T2c. In what content areas did you use LDC modules and/or mini-tasks? 


T2d. In what grades did you use LDC modules and/or mini-tasks? 


89 


T3. Prior to the current school year (2017-18), did you have any experience with LDC? 


(n = 245) 
Yes 61 teachers 24.9% 
No y Skip to T5a 184 teachers 75.1% 


T4. How many of the following did you teach prior to the current school year (2017-18)? 


| | LDC modules n= 60, Mean = 1.67, Range: 0-10 


LDC mini-tasks, outside of modules n=60, Mean =1.95, Range: 0-10 


Professional Learning Community and Teacher 
Collaboration 


T5a. Did you participate this year in a Professional Learning Community (PLC) at least partly 
focused on implementing LDC in your school? 


(n = 204) 
Yes y Skip to T6 196 teachers 96.1% 
No 8 teachers 3.9% 


T5b. Did you use any LDC tools in your instructional planning or classroom instruction this 


year? 
(n = 8) 
Yes <> Skip to 5d 8 teachers (100%) 
No 0 teachers (0%) 


T5c. Why did you choose not to use any LDC tools in your instructional planning or 
classroom instruction this year? 


[Survey ends here for respondents answering question T5c] 
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T5d. What LDC tools did you use during the current school year? Select all that apply. 


LDC online courses 


Other (please specify) 


[Survey ends here for respondents answering question T5d] 


T6. About how often did your LDC PLC meet? 


Less than once a month 


Once a month 


Every other week 
Once a week y Skip to T8 


Twice a week or more often <> Skip to T8 


(n=8) 
CoreTools online platform to access existing modules or mini-tasks 8 teachers (100%) 
CoreTools online platform to design modules or mini-tasks 4 teachers (50.0%) 


2 teachers (25.0%) 


Modules or mini-tasks given to me by other teachers in my school 2 teachers (25.0%) 


0 teachers (0%) 


(n = 236) 

3 teachers (1.3%) 

53 teachers (22.5%) 
151 teachers (64.0%) 
28 teachers (11.9%) 
1 teacher (0.4%) 


T7. What were the primary barriers preventing your LDC PLC from meeting weekly? Select 


all that apply. 


PLC time was not protected. 


School administrator did not make it a priority. 


Not enough teachers participated. 


PLC members had other priorities that compete with 


Other (please specify) 
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(n = 207) 
38 teachers (18.4%) 


PLC members had limited interest in attending meetings. 23 teachers (11.1%) 


20 teachers (9.7%) 


Teacher Leader did not provide sufficient organizational support. 6 teachers (2.9%) 


17 teachers (8.2%) 
LDC participation. 112 teachers (54.1%) 
72 teachers (34.8%) 


T8. About how often did you have informal discussions (as opposed to scheduled meetings) 
about LDC with teachers in your LDC PLC? 


(n = 236) 


Less than once a month 


Once a month 


Every other week 


Once a week 


Twice a week or more 


31 teachers (13.1%) 
43 teachers (18.2%) 
54 teachers (22.9%) 
71 teachers (30.1%) 
37 teachers (15.7%) 


T9. On average, how long did your school’s LDC PLC meetings typically last? 


(n = 236) 


Less than 45 minutes 


45 to 59 minutes 


60 to 74 minutes 


75 minutes or more 


Teacher Training and Support 


11 teachers (4.7%) 
134 teachers (56.8%) 
86 teachers (36.4%) 
5 teachers (2.1%) 


T10. How effective was your LDC PLC in the following areas? 


instructional plans. (n = 236) 


Not A little Moderately Very 
effective effective effective effective 

Creating an environment in which 
teachers were comfortable working 5 (2.1%) 9 (3.8%) 107 (45.3%) | 115 (48.7%) 
together (n = 236) 
Fostering an environment where 
teachers shared their instructional 4 (1.7%) 25 (10.6%) | 100(42.4%) | 107 (45.3%) 
plans with colleagues (n = 236) 
Allowi to sh tudent k 

eee ae gee 7 (3.0%) | 33(14.0%) | 99(41.9%) | 97 (41.1%) 
(n = 236) 
Helping teachers to improve their LDC 10 (4.2%) 39 (16.5%) 108 (45.8%) 79 (33.5%) 
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T11. How would you rate each of the following aspects of the online course material (in the 
Learn tab in LDC CoreTools) that your coach used or directed you to use? 


Poor Fair Good Excellent 


Clarity of information presented (n = 236) 6 (2.5%) 47 (19.9%) | 138 (58.5%) | 45 (19.1%) 


Relevance of information presented 3 (1.3%) 55 (23.3%) | 116 (49.2%) | 62 (26.3%) 


(n = 236) 

Ease of use (n = 236) 20 (8.5%) | 88 (37.3%) | 100(42.4%) | 28 (11.9%) 
Usefulness of resource documents (e.g., 

LDC Pitfall Checklist, CCSS Mental 14 (5.9%) | 48(20.3%) | 126 (53.4%) | 48 (20.3%) 
Markers, etc.) (n = 236) 

Usefulness of videos (n = 236) 17 (7.2%) 83 (35.2%) | 111 (47.0%) | 25 (10.6%) 


Degree to which course material helped 
teachers to create and/or adapt LDC 9 (3.8%) 68 (28.8%) | 115 (48.7%) | 44 (18.6%) 
modules (n = 236) 


Opportunity to extend learning when 


0, Le) Le) 0, 
needed or desired (ne 250) 11 (4.7%) | 61(25.8%) | 111 (47.0%) | 53 (22.5%) 


T12a. Overall, were you able to get the feedback and support you needed from your LDC 
coach (through written feedback in LDC CoreTools, or coaching and modeling in your 
LDC PLCs) to plan, teach, reflect on, and revise LDC modules)? 


(n = 236) 
Yes 226 teachers (95.8%) 
No 10 teachers (4.2%) 


T12b. Did your LDC coach provide written feedback on your module(s) in LDC CoreTools ina 
timely manner? 


(n = 236) 
Yes 219 teachers (92.8%) 
No 17 teachers (7.2%) 
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T13. Outside of the PLC meetings with your LDC coach, please indicate whether you used 
each of the following types of coach support, and how helpful you found these types 


of support. 


Did not use 


Used 


Not 
helpful 


A little 
helpful 


Moderately 
helpful 


Very 
helpful 


Written feedback in LDC 
CoreTools from your LDC 
coach (in the comments 
areas and/or via the 
teacher work rubric) 

(n = 236) 


49 (20.8%) 


5 (2.1%) 


41 (17.4%) 


82 (34.7%) 


59 (25.0%) 


One-on-one Zoom video 
conference and/or call 
with your LDC coach 

(n = 236) 


51 (21.6%) 


10 (4.2%) 


15 (6.4%) 


60 (25.4%) 


100 (42.4%) 


Email or phone 
communication with your 
LDC coach (n = 236) 


75 (31.8%) 


5 (2.1%) 


23 (9.7%) 


58 (24.6%) 


75 (31.8%) 


Other Please specify: 
(n = 236) 


186 (78.8%) 


2 (0.8%) 


3 (1.3%) 


15 (6.4%) 


30 (12.7%) 
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Module Creation 


T14. During the current school year (2017-18), how many LDC modules did you individually 
or collaboratively adapt from existing modules (e.g., modules you created in a prior 
year and/or modules found in the LDC Library in CoreTools)? 


| | Adapted modules n=196, Mean = 1.94, Range: 0-4 


T15. During the current school year (2017-18), how many LDC modules did you create, 
either individually or with colleague(s)? Only include modules built from scratch, not 
those adapted from existing modules in the LDC library. 


| | New modules n=196, Mean = 0.74, Range: 0-12 


T16. How did members of your PLC collaborate to create LDC modules? 
Check all that apply. 


(n = 236) 
Modules were created by individual teachers. 60 teachers (25.4%) 
Modules were created by teams of two or more teachers. 148 teachers (62.7%) 
Modules were created by the PLC as a whole. 43 teachers (18.2%) 
Other (please specify) __ _ _— 37 teachers (15.7%) 
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T17. Please indicate to what extent you were able to do each of the following when 


creating LDC modules. 


Not at all 


A little bit 


A moderate 
extent 


A great 
extent 


Select focus standards for a writing 
assignment (n = 236) 


3 (1.3%) 


15 (6.4%) 


81 (34.3%) 


137 (58.1%) 


Create a standards-driven writing 
assignment (n = 236) 


5 (2.1%) 


15 (6.4%) 


86 (36.4%) 


130 (55.1%) 


Select high quality, complex texts and 
other materials to engage students in 
deeper learning (n = 236) 


7 (3.0%) 


38 (16.1%) 


110 (46.6%) 


81 (34.3%) 


Identify the skills students need to 
develop to complete a writing 
assignment (n = 236) 


6 (2.5%) 


28 (11.9%) 


98 (41.5%) 


104 (44.1%) 


Create daily lessons to teach the skills 
a student needs to complete a writing 
assignment (n = 236) 


19 (8.1%) 


AA (18.6%) 


109 (46.2%) 


64 (27.1%) 


Differentiate instruction by 
incorporating multiple ways of 
thinking, various levels of complexity, 
and multiple modalities (n = 236) 


15 (6.4%) 


51 (21.6%) 


111 (47.0%) 


59 (25.0%) 


Plan for a variety of methods to 
assess student progress (e.g., rubrics 
and/or mini-task scoring guides) 

(n = 236) 


18 (7.6%) 


50 (21.2%) 


108 (45.8%) 


60 (25.4%) 


Assess the quality of writing 
assignments and/or instructional 
plans using Peer Review/Curriculum 
Alignment Rubric (e.g. Task Pitfalls 
Checklist, rubric indicators) (n = 236) 


14 (5.9%) 


50 (21.2%) 


99 (41.9%) 


73 (30.9%) 


Make a writing assignment relevant 
and engaging for students (n = 236) 


9 (3.8%) 


24 (10.2%) 


109 (46.2%) 


94 (39.8%) 


Classroom Implementation 


T18. How many total LDC modules did you teach during the current school year (2017-18)? 


a Modules n= 236, Mean = 2.19, Range: 0-20 
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T19. Outside of modules, approximately how many individual LDC mini-tasks did you teach 
during the current school year (2017-18)? 


a Mini-tasks n= 236, Mean = 4.06, Range: 0-20 


T20. Please indicate to what extent you were able to do each of the following activities 


when teaching LDC modules. 


instruction (n = 236) 


A A 
Not at all A little bit moderate Breet 
extent extent 

Engage students in understanding the 

6 (2.57 31 (13.19 99 (41.99 100 (42.49 
assignment and its rubric (n = 236) tel ( a ss eile ( me 
Engage students in accessing complex 
text for the purpose of the assignment 4 (1.7%) 27 (11.4%) | 104 (44.1%) | 101 (42.8%) 
(n = 236) 
Systematically collect information 

6 (2.57 43 (18.29 116 (49.29 71 (30.19 
about students’ progress (n = 236) Lee ( a a ( mi 
PrOvIDS Teach aen to spent USIne 11 (4.7%) | 39(16.5%) | 113 (47.9%) | 73 (30.9%) 
assignment rubrics (n = 236) 
Locate evidence of standards in final 
student work on the writing 6 (2.5%) 24 (10.2%) | 107 (45.3%) | 99 (41.9%) 
assignment (n = 236) 
Use evidence of student progress on 
standards to modify subsequent 7 (3.0%) 39 (16.5%) | 110(46.6%) | 80 (33.9%) 


T21. Toward the beginning of the school year, did you ’find and teach” a module from CoreTools? 


Yes 


No y Skip to T24 


(n = 236) 
181 teachers (76.7%) 


55 teachers (23.3%) 


T22. What was the name of the Find and Teach module? 


T23. Did you make any adjustments to the Find and Teach module? 


(n = 181) 
Yes 145 teachers (80.1%) 
No 36 teachers (19.9%) 
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T24. What module did you adapt, refine, and/or develop most during the current school year 
(2017-18)? This module is typically one you worked on after the Find and Teach module. 


T25. Which of these statements best describes how you created the module named in the 
previous question? 


(n = 236) 


| created a module from a template in CoreTools. 63 teachers (26.7%) 


| found and adjusted another teacher’s module from the LDC Library 173 teachers (73.3%) 
in CoreTools. 


T26. Did you teach this module in your classroom? 


(n = 236) 
Yes, | have already taught this module this year. 219 teachers (92.8%) 
No, but | plan to teach this module before the end of the 2017-18 7 teachers (3.0%) 
school year. 
No, but | plan to teach this module during next school year. 6 teachers (2.5%) 
No. | do not currently have plans to teach this module in my classroom. 4 teachers (1.7%) 


Module Peer Review 


T27. Did you attend a Peer Review/Curriculum Alignment Workshop this school year? (Y/N) 


(n = 234) 
Yes 68 teachers (29.1%) 
No 166 teachers (70.9%) 
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T28. How many modules did you submit online for LDC National Peer Review during the 
current school year (2017-18)? 


| | Modules ifnone, — skip to T30 n = 236, Mean = 0.51, Range: 0-4 


T29. How helpful did you find the National Peer Review process in improving the quality of 
your module? 


(n = 66) 
Not helpful 12 teachers (18.2%) 
A little helpful 18 teachers (27.3%) 
Moderately helpful 25 teachers (37.9%) 
Very helpful 11 teachers (16.7%) 


y | Impact on Teacher Practice and Learning 


T30. Between the beginning and end of this year’s work with LDC, please indicate how 
much your skills have improved in the following areas: 


Not at all A little Moderately | A great deal 


Selecting focus standards for a 


0, 0, 0, 0, 
Specie caenremen eames 7 11 (4.7%) 28 (11.9%) | 108 (45.8%) | 89 (37.7%) 


Creating standards-driven writing 


0, 0, 0 0, 
senna 12 (5.1%) 33 (14.0%) | 105 (44.5%) | 86 (36.4%) 


Identifying the skills students need to 
develop to complete a writing 12 (5.1%) 33 (14.0%) 104 (44.1%) 87 (36.9%) 
assignment (n = 236) 


Creating daily lessons to teach the 
skills students need to complete a 19 (8.1%) 39 (16.5%) 121 (51.3%) 57 (24.2%) 
writing assignment (n = 236) 


Systematically collecting information 


0, 0, 0, 0, 
Seated ee 16 (6.8%) 43 (18.2%) | 119 (50.4%) | 58 (24.6%) 


Identifying patterns of student 
understandings or misconceptions 18 (7.6%) 42 (17.8%) 114 (48.3%) 62 (26.3%) 
(n = 236) 


Using evidence of student progress 
on standards to modify subsequent 13 (5.5%) 41 (17.4%) 105 (44.5%) 77 (32.6%) 
instruction (n = 236) 
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T31. Please indicate the degree to which you agree or disagree with the statements below. 


I l 
Strone Y Disagree Agree stronely 
disagree agree 
Participating in LDC raised my 
expectations for students’ writing. 11 (4.7%) 37 (15.7%) | 121 (51.3%) | 67 (28.4%) 
(n = 236) 
Using LDC modules became an 
important part of my instructional 16 (6.8%) 68 (28.8%) | 116 (49.2%) | 36 (15.3%) 


practice. (n = 236) 


Implementing LDC helped me 
incorporate my state’s College- and 
Career-Ready Standards into my 
instruction. (n = 236) 


11 (4.7%) | 43(18.2%) | 137(58.1%) | 45 (19.1%) 


LDC helped me incorporate writing 
assignments into my existing 13 (5.5%) 32 (13.6%) | 127 (53.8%) | 64 (27.1%) 
curriculum. (n = 236) 


lam more likely to collaborate with 
other teachers on designing instruction 
after participating in our LDC 11 (4.7%) 38 (16.1%) | 128 (54.2%) | 59 (25.0%) 
Professional Learning Community. 
(n = 236) 


LDC helped me improve on my teacher 


0, 0, Le) ie) 
Sy alictionastines ees 19 (8.1%) | 80(33.9%) | 99(41.9%) | 38 (16.1%) 


Participating in LDC helped me develop 
working relationships with teachers in 
different grades and/or subjects. 

(n = 236) 


13 (5.5%) | 48(20.3%) | 117(49.6%) | 58 (24.6%) 


| shared my LDC work with colleagues 


0, 0, 0 0, 
outside of the LDC PLC. (n = 236) 37 (15.7%) | 79 (33.5%) | 81(34.3%) | 39 (16.5%) 
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Impact on Student Learning 


T32. Please indicate to what extent LDC had a positive effect on students in the following 


areas. 

Not at all A little Moderately | A great deal 
Reading skills (n = 236) 21 (8.9%) 52 (22.0%) | 110(46.6%) | 53 (22.5%) 
Content knowledge (n = 236) 12 (5.1%) 38 (16.1%) 94 (39.8%) 92 (39.0%) 
Ability to complete writing a ‘ : ; 
seclenmentotn as 12 (5.1%) 34 (14.4%) | 104(44.1%) | 86 (36.4%) 
Quality of students’ writing (n = 236) 15 (6.4%) 39 (16.5%) | 102 (43.2%) | 80 (33.9%) 
Calica snacareerr edgy skis 22 (9.3%) | 45(19.1%) | 101 (42.8%) | 68 (28.8%) 
(n = 236) 
Capacity to analyze and understand 
the components of a writing 14 (5.9%) 41 (17.4%) 102 (43.2%) | 79 (33.5%) 
assignment (n = 236) 
Speaking and listening skills (n = 236) 23 (9.7%) 51 (21.6%) | 114(48.3%) | 48 (20.3%) 
Overall literacy performance (n = 236) 17 (7.2%) 44 (18.6%) 116 (49.2%) | 59 (25.0%) 
Perormience Gmtassessinents 22 (9.3%) | 50(21.2%) | 118(50.0%) | 46 (19.5%) 


throughout the school year (n = 236) 
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Teacher Leader Support 


The following question refers to the LDC teacher leader in your school. This is the teacher leading your 


Professional Learning Community work. 


T33. Please indicate the degree to which you agree or disagree with the following 


statements. 
suonely Disagree Agree strongly 
disagree agree 
Our school’s LDC teacher leader 
effectively supported our Professional 5 (2.6%) 2 (1.0%) 93 (47.4%) | 96 (49.0%) 
Learning Community meetings. (n = 196) 
When | had questions about LDC, | felt 
comfortable approaching our school’s 4 (2.0%) 4 (2.0%) 79 (40.3%) | 109 (55.6%) 
teacher leader. (n = 196) 
Our teacher leader helped teachers align 
LDC to broader school instructional goals. 6 (3.1%) 14 (7.1%) 84 (42.9%) | 92 (46.9%) 
(n = 196) 
Our teacher leader offered useful 
feedback for the design and revision of 6 (3.1%) 16 (8.2%) 86 (43.9%) | 88 (44.9%) 
LDC modules. (n = 196) 
Our teacher leader was effective in 
inviting teachers to join the LDC 4 (2.0%) 13 (6.6%) 88 (44.9%) | 91 (46.4%) 


initiative. (n = 196) 
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Ke) School Administrator Support 


The following questions refer to the school administrator who oversees the LDC project at your school. 


T34. What proportion of PLC meetings focused on LDC did your school administrator attend? 


(n=236) 
Less than one quarter of LDC PLCs 72 teachers (30.5%) 
About one quarter of LDC PLCs 31 teachers (13.1%) 
About one half of LDC PLCs 34 teachers (14.4%) 
About three quarters of LDC PLCs 21 teachers (8.9%) 
More than three quarters of LDC PLCs 78 teachers (33.1%) 


T35. How many times did your school administrator observe you teach an LDC mini-task 
during the current school year (2017-18)? 


(n = 236) 
0 times 119 teachers (50.4%) 
1 time 57 teachers (24.2%) 
2 times 33 teachers (14.0%) 
3 or more times 27 teachers (11.4%) 
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T36. Please indicate the degree to which you agree or disagree with the following 


statements. 
My school administrator... SHenely Disagree Agree SHengly 
disagree agree 

pias tip Une rstandilie Ob ERE. 11 (4.7%) | 52 (22.0%) | 129(54.7%) | 44 (18.6%) 
(n = 236) 
allocated resources such as teacher time, 
payment, administrator time, support ‘ : : 
staff, sub coverage, etc., to ensure the a eh || Sener. || See ee 
LDC team could meet. (n = 236) 
encouraged teachers to participate in 7 (3.0%) 19 (8.1%) 127 (53.8%) | 83 (35.2%) 


LDC. (n = 236) 


expressed concerns that implementing 
LDC is taking time away from other 
instructional priorities. (n = 236) 


52 (22.0%) 


97 (41.1%) 


55 (23.3%) 


32 (13.6%) 


communicated how using LDC’s tools 
supported specific school initiatives 
and/or goals. (n = 236) 


14 (5.9%) 


50 (21.2%) 


122 (51.7%) 


50 (21.2%) 


provided me with feedback about my 
LDC planning and/or instruction. 
(n = 236) 


37 (15.7%) 


93 (39.4%) 


78 (33.1%) 


28 (11.9%) 


made formative assessment a priority at 
my school. (n = 236) 


21 (8.9%) 


54 (22.9%) 


118 (50.0%) 


43 (18.2%) 


used LDC to implement standards-driven 
assignments within existing curriculum. 
(n = 236) 


25 (10.6%) 


57 (24.2%) 


108 (45.8%) 


A6 (19.5%) 
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Teacher Leadership Role 


T37. Please indicate the degree to which you agree or disagree with the following 
statements about your role in your school’s LDC implementation. 


future years. (n=196) 


I l 
Stone y Disagree Agree strenely 
disagree agree 
| was involved in setting instructional goals 16 (8.2%) | 58 (29.6%) | 96 (49.0%) | 26 (13.3%) 
for the LDC work at my school. (n=196) es is ie we 
| was involved in discussions about how to 
expand LDC implementation at my school in | 23 (11.7%) | 73 (37.2%) | 77 (39.3%) | 23 (11.7%) 


| had the opportunity to work with our LDC 
teacher leader and our administrator to 
help shape LDC implementation. (n=196) 


21 (10.7%) 


76 (38.3%) 


76 (38.8%) 


23 (11.7%) 


| am interested in learning more about how 
to lead LDC implementation at my school by 
facilitating with the virtual coach, providing 
feedback to my peers, etc. (n=196) 


33 (16.8%) 


61 (31.1%) 


77 (39.3%) 


25 (12.8%) 
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12 Facilitators and Barriers 


T38. Please indicate the degree to which you agree or disagree with the following 


statements. 
strongly Disagree Agree etrane 
disagree agree 
My LDC PLC was given sufficient time 17 (7.2%) 64 (27.1%) | 112 (47.5%) | 43 (18.2%) 
to meet. (n = 236) 
| felt adequately prepared to 
effectively implement LDC modules in 11 (4.7%) 54 (22.9%) | 135 (57.2%) | 36 (15.3%) 
my classroom. (n = 236) 
It was challenging to find content-rich 
reading materials for the LDC modules | 17 (7.2%) 79 (33.5%) | 102 (43.2%) | 38 (16.1%) 
developed. (n = 236) 
My school had adequate technology to ‘ " ‘ 4 
suinport teachers’ Useor tbe, towel 6 (2.5%) 24 (10.2%) | 134(56.8%) | 72 (30.5%) 
It was easy to find and adapt LDC mini- 9 (3.8%) 54 (22.9%) | 134 (56.8%) | 39 (16.5%) 


tasks for use in my classroom. (n = 236) 


ie} Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 


e CoreTools online platform 


e LDC online courses in the “Learn” section of CoreTools 


e Virtual coaching 


O Zoom meetings, written feedback on teacher work in LDC CoreTools, emails, etc. 


e  In-person coaching 


O Summer training, in-person support visits from LDC and District Lead, in-person 


professional development opportunities, etc. 
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T39. What supports did you find the most useful and why? 


T40. What supports were not helpful and why? 


T41. In what ways could LDC implementation be improved in your school in the future? 
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Appendix B: 
Teacher Leader Survey and Responses 


2017-2018 
LDC Teacher Leader Survey 


LDC Participation 


TL1. Prior to the current school year (2017-18), did you have any experience with LDC? 


(n = 48) 
Yes 15 liaisons (31.3%) 
No i+ Skip toTL3 33 liaisons (68.8%) 


TL2. How many of the following did you teach prior to the current school year (2017-18)? 


| | LDC modules n=15, Mean = 1.73, Range: 0-7 


LDC mini-tasks, outside of modules n=15, Mean = 1.73, Range: 0-10 
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Professional Learning Community and Teacher 


Collaboration 


The following questions involve the LDC Professional Learning Community (PLC) that you are leading. 


TL3. About how often did your LDC PLC meet? 


Less than once a month 


Once a month 


Every other week 


Once a week y Skip to TL5 
Twice a week or more <> Skip to TLS 
often 


(n = 48) 

0 liaisons (0.0%) 

5 liaisons (10.4%) 
35 liaisons (72.9%) 
8 liaisons (16.7%) 


0 liaisons (0.0%) 


TL4. What were the primary barriers preventing your LDC PLC from meeting weekly? Select 


all that apply. 

(n = 40) 
PLC time was not protected. 9 liaisons (22.5%) 
PLC members had limited interest in attending meetings. 6 liaisons (15.0%) 
School administrator did not make it a priority. 6 liaisons (15.0%) 
| was unable to provide sufficient organizational support. 0 liaisons (0.0%) 


Not enough teachers participated. 


Other (please specify) 


3 liaisons (7.5%) 


PLC members had other priorities that competed with LDC participation. 20 liaisons (50.0%) 


0 liaisons (0.0%) 


TL5. About how often did you have informal discussions (as opposed to scheduled 
meetings) about LDC with teachers in your LDC PLC? 


(n = 48) 
Less than once a month 5 liaisons (10.4%) 
Once a month 8 liaisons (16.7%) 
Every other week 15 liaisons (31.3%) 
Once a week 13 liaisons (27.1%) 
Twice a week or more 7 liaisons (14.6%) 
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TL6. On average how long did your school’s LDC PLC meetings typically last? 


(n = 48) 


45 to 59 minutes 


60 to 74 minutes 


Teacher Training and Support 


Less than 45 minutes 2 liaisons (4.2%) 
29 liaisons (60.4%) 
17 liaisons (35.4%) 


75 minutes or more 0 liaisons (0.0%) 


TL7. How effective was your LDC PLC in the following areas? 


LDC instructional plans. (n = 47) 


Not A little Moderately Very 

effective effective effective effective 
Creating an environment in which 
teachers are comfortable working 0 (0.0%) 0 (0.0%) 17 (36.2%) 30 (63.8%) 
together (n = 47) 
Fostering an environment where 
teachers share their instructional plans 0 (0.0%) 3 (6.4%) 16 (34.0%) 28 (59.6%) 
with colleagues (n = 47) 
Allowi to sh tudent k 
nes Sit ents ec soe pt 0 (0.0%) 4(8.5%) | 18(38.3%) | 25 (53.2%) 
Helping teachers learn to improve their 1 (2.1%) 4 (8.5%) 20 (42.6%) 22 (46.8%) 
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TL8. How would you rate each of the following aspects of the online course material (in the 
Learn tab in LDC CoreTools) that your coach used or directed you to use? 


or desired (n = 47) 


Poor Fair Good Excellent 
Clarity of information presented (n = 47) 0 (0.0%) 7 (14.9%) | 28(59.6%) | 12 (25.5%) 
Relevance of information presented (n = 47) 0 (0.0%) 7 (14.9%) | 22 (46.8%) | 18 (38.3%) 
Ease of use (n = 47) 1 (2.1%) | 20(42.6%) | 18 (38.3%) | 8 (17.0%) 
Usefulness of resource documents (e.g., LDC 
Pitfall Checklist, CCSS Mental Markers, etc.) 0 (0.0%) 8 (17.0%) | 22 (46.8%) | 17 (36.2%) 
(n = 47) 
Usefulness of videos (n = 47) 0 (0.0%) 12 (25.5%) | 28 (59.6%) | 7 (14.9%) 
Degree to which course material helped 
teachers to create and/or adapt LDC modules 1 (2.1%) 6 (12.8%) | 25 (53.2%) | 15 (31.9%) 
(n = 47) 
Opportunity to extend learning when needed 2 (4.3%) 6 (12.8%) | 22 (46.8%) | 17 (36.2%) 


TL9a. Overall, were you able to get the feedback and support you needed from your LDC 
coach (through written feedback in LDC CoreTools, or coaching and modeling in your 


LDC PLCs) to plan, teach, reflect on, and revise LDC modules? 


(n = 47) 
Yes 47 liaisons (100%) 
No O liaisons (0.0%) 


TL9b. Did your LDC coach provide written feedback on your module(s) in LDC CoreTools ina 


timely manner? 


(n = 47) 
Yes 46 liaisons (97.9%) 
No 1 liaison (2.1%) 
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TL10. Outside of the PLC meetings with your LDC coach, please indicate whether you used 
each of the following types of coach support, and how helpful you found these types 


of support. 
Used 
Did not 
se Not A little Moderatel Very 

helpful helpful y helpful helpful 
Written feedback in LDC 
CoreTools from your LDC coach 
(in the comments areas and/or 6 (12.8%) 0 (0.0%) 2 (4.3%) | 14(29.8%) | 25 (53.2%) 
via the teacher work rubric) 
(n = 47) 
One-on-one Zoom video 
conference and/or call with 1 (2.1%) 0 (0.0%) 1(2.1%) | 14(29.8%) | 31 (66.0%) 
your LDC coach (n = 47) 
Email or phone communication ‘ ’ . ‘ 
with your LDC coach (<a7) 1 (2.1%) 0(0.0%) | 0(0.0%) | 9(19.1%) | 37 (78.7%) 
Other (please specify) (n = 47) 35 (74.5%) 0 (0.0%) 0 (0.0%) 4 (8.5%) 8 (17.0%) 


TL11. How many in-person and/or online LDC professional development offerings for school 
administrators and teacher leaders did you attend during the current school (e.g., Summer 
in-person launch days, Quarterly in-person teacher leader meetings, LDC monthly virtual 
coach meetings)? 


| | Professional development offerings 


n=47, Mean = 5.85, Range: 0-20 
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Support to Teacher Leader from LDC Coach 


TL12. Please indicate the degree to which you agree or disagree with the following 
statements. 


Strongly Disagree gree Strongly 


disagree agree AYA 


| was able to reach my LDC coach 
if | had any questions about LDC. 0 (0.0%) 2 (4.3%) 6 (12.8%) | 38(80.9%) | 1(2.1%) 
(n = 47) 


LDC provided adequate technical 
support for issues with the 
CoreTools online platform. 

(n = 47) 


0(0.0%) | 0(0.0%) | 21(44.7%) | 19 (40.4%) | 7 (14.9%) 


LDC offered sufficient 
professional development 
opportunities for me to lead the 
initiative in my school. (n = 47) 


1 (2.1%) | 4(8.5%) | 22 (46.8%) | 20(42.6%) | 0 (0.0%) 


LDC coaches were able to 
connect me with additional 0 (0.0%) 0(0.0%) | 20(42.6%) | 27(57.4%) | 0(0.0%) 
resources when needed. (n = 47) 


It was challenging to coordinate 
with our LDC coach on how to 
structure Professional Learning 
Community time. (n = 47) 


14 (29.8%) | 14(29.8%) | 9(19.1%) | 8(17.0%) | 2 (4.3%) 


When | reached out to our LDC 
coach, he or she responded 0 (0.0%) 2 (2.1%) | 12 (25.5%) | 34(72.3%) | 0 (0.0%) 
quickly. (n = 47) 


Our LDC coach was easy to work 


0, 0, 0, 0, 0, 
aeceena 0(0.0%) | 1(2.1%) | 7(14.9%) | 38(80.9%) | 1 (2.1%) 


Our LDC coach was 
knowledgeable and provided 0 (0.0%) 1 (2.1%) 8 (17.0%) | 36(76.6%) | 2 (4.3%) 
high quality guidance. (n = 47) 
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Module Creation 


TL13. During the current school year (2017-18), how many LDC modules did your PLC 
individually or collaboratively adapt from existing modules (e.g., modules created in 
a prior year and/or modules from the LDC Library in CoreTools)? 


| | Adapted modules n= 47, Mean = 2.47, Range: 1-10 


TL14. During the current school year (2017-18), how many LDC modules did your PLC create 
(either individually or in a group)? Only include modules built from scratch, not those 
adapted from existing modules in the LDC library. 


| | New modules n= 47, Mean = 0.96, Range: 0-6 


TL15. How did members of your PLC collaborate to create LDC modules? Check all that 


apply. 
(n =47) 
Modules were created by individual teachers. 10 liaisons (21.3%) 
Modules were created by teams of two or more teachers. 27 liaisons (57.4%) 
Modules were created by the PLC as a whole. 11 liaisons (23.4%) 
Other (please specify) es 9 liaisons (19.1%) 
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TL16. Please indicate to what extent you were able to do each of the following when 
creating LDC modules. 


aoe To a great 
Not at all A little bit moderate 6 
extent 
extent 

Select focus standards for a writing 0 (0.0%) 1 (2.1%) 15 (31.9%) 31 (66.0%) 
assignment (n = 47) 
Create a standards-driven writing 

0 (0.0%) 1 (2.1%) 17 (36.2%) 29 (61.7%) 


assignment (n = 47) 


Select high quality, complex texts and 
other materials to engage students in | 1 (2.1%) 4 (8.5%) 22 (46.8%) 20 (42.6%) 
deeper learning (n = 47) 


Identify the skills students need to 
develop to complete a writing 0 (0.0%) 5 (10.6%) 17 (36.2%) 25 (53.2%) 
assignment (n = 47) 


Create daily lessons to teach the 
skills a student needs to complete a 0 (0.0%) 10 (21.3%) 19 (40.4%) 18 (38.3%) 
writing assignment (n = 47) 


Differentiate instruction by 
incorporating multiple ways of 


fo} fo} fo} fo} 
thinking, various levels of complexity, eles od aie PAGE TRL AA 
and multiple modalities. (n = 47) 
Plan for a variety of methods to 
assess student progress (e.g., rubrics 2 (4.3%) 9 (19.1%) 22 (46.8%) 14 (29.8%) 


and/or mini-task scoring guides) 
(n= 47) 


Assess the quality of writing 
assignments and/or instructional 
plans using Peer Review/Curriculum 0 (0.0%) 9 (19.1%) 20 (42.6%) 18 (38.3%) 
Alignment Rubric (e.g. Task Pitfalls 
Checklist, rubric indicators) (n = 47) 


Make a writing assignment relevant 


0 0, 0, Le) 
sracnesene adc 0 (0.0%) 4 (8.5%) 20 (42.6%) 23 (48.9%) 
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Impact on Student Learning 


TL17.Please indicate to what extent LDC had a positive effect on students in the following 


areas. 
Not at all A little Moderately | A great deal 

Reading skills (n = 47) 2 (4.3%) 8 (17.0%) 18 (38.3%) 19 (40.3%) 
Content knowledge (n = 47) 1 (2.1%) 4 (8.5%) 20 (42.6%) 22 (46.8%) 
Ability to complete writing ‘ ‘i . : 
selenite te 1 (2.1%) A (8.5%) 17 (36.2%) | 25 (53.2%) 
Quality of students’ writing (n = 47) 1 (2.1%) 3 (6.4%) 21 (44.7%) 22 (46.8%) 
College and career ready skills (n = 47) 1 (2.1%) 7 (14.9%) 22 (46.8%) 17 (36.2%) 
Capacity to analyze and understand 
the components of a writing 0 (0.0%) 5 (10.6%) 23 (48.9%) 19 (40.4%) 
assignment (n = 47) 
Speaking and listening skills (n = 47) 3 (6.4%) 7 (14.9%) 22 (46.8%) 15 (31.9%) 
Overall literacy performance (n = 47) 1 (2.1%) 7 (14.9%) 20 (42.6%) 19 (40.4%) 
Perf t 

Sal Ue aN E uel HPN 2 (4.3%) 6 (12.8%) | 23(48.9%) | 16 (34.0%) 


throughout the school year (n = 47) 
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y | School Administrator Support 


The following questions refer to the school administrator who oversees the LDC project at your school. 


TL18. What proportion of PLC meetings focused on LDC did your school administrator attend? 


(n = 47) 
Less than one quarter of LDC PLCs 14 liaisons (29.8%) 
About one quarter of LDC PLCs 7 liaisons (14.9%) 
About one half of LDC PLCs 8 liaisons (17.0%) 
About three quarters of LDC PLCs 3 liaisons (6.4%) 
More than three quarters of LDC PLCs 15 liaisons (31.9%) 


117 


TL19. Please indicate the degree to which you agree or disagree with the following 


statements. 

My school administrator... Streneyy Disagree Agree suongly 
disagree agree 

had a firm understanding of LDC. (n = 47) 1 (2.1%) 11 (23.4%) | 24 (51.1%) | 11 (23.4%) 
allocated resources such as teacher time, 
payment, administrator time, support staff, - ‘ r ‘ 
sub coverage, etc., to ensure the LDC team ae) AQ (2L. 3%) | TPRG-2%) | TS 140.4%) 
could meet. (n = 47) 
encouraged teachers to participate in LDC. 1 (2.1%) 4 (8.5%) 21 (44.7%) | 21 (44.7%) 


(n = 47) 


expressed concerns that implementing LDC is 


taking time away from other instructional 
priorities. (n = 47) 


14 (29.8%) 


25 (53.2%) 


4 (8.5%) 


4 (8.5%) 


communicated how using LDC’s tools 


supported specific school initiatives and/or 3 (6.4%) 10 (21.3%) | 22 (46.8%) | 12 (25.5%) 
goals. (n = 47) 

provided me with feedback about my LDC ; ‘ ‘ ‘ 
planning and/or instruction. (n = 47) oe) See) | Leen oeey | LEI 
made formative assessment a priority at my 2 (4.3%) 14 (29.8%) | 21 (44.7%) | 10 (21.3%) 
school. (n = 47) 

Used LDC to implement standards-driven 

assignments within existing curriculum. 3 (6.4%) 14 (29.8%) | 18 (38.3%) | 12 (25.5%) 


(n = 47) 
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Teacher Leader Leadership Role 


TL20. Please indicate the degree to which you agree or disagree with the following 
statements about your role in leading your school’s LDC implementation. 


Sirenely Disagree Agree SuCnely 
disagree agree 

| met regularly with my school administrator 

to make planning decisions around LDC. 3 (6.4%) 12 (25.5%) | 24(51.1%) | 8 (17.0%) 


(n =47) 


| was involved in discussions about 
differentiating LDC implementation to meet 2 (4.3%) 6 (12.8%) | 29 (61.7%) | 10 (21.3%) 
teacher learning needs. (n = 47) 


| was involved in discussions about how to 
expand LDC implementation at my school in 1 (2.1%) 10 (21.3%) | 24(51.1%) | 12 (25.5%) 
future years. (n = 47) 


My role as an LDC teacher leader allowed me 
to effectively advocate for additional 1 (2.1%) 13 (27.7%) | 22 (46.8%) | 11 (23.4%) 
resources on my campus. (n = 47) 


| was involved in adjusting the problems of 
practice that my school targeted with the 1 (2.1%) 12 (25.5%) | 25 (53.2%) | 9 (19.1%) 
LDC work. (n = 47) 


| met regularly with my LDC coach to manage 


0 0, Le) 0, 
neice work an tail 1 (2.1%) 1(2.1%) | 27 (57.4%) | 18 (38.3%) 


| feel that my position as an LDC teacher 
leader allowed me to build my capacity as an 
instructional leader among my colleagues. 

(n =47) 


1 (2.1%) 2 (4.3%) | 30(63.8%) | 14 (29.8%) 


| am confident that | can lead our LDC PLC in 
the future without the assistance of an LDC 3 (6.4%) 20 (42.6%) | 20(42.6%) | 4(8.5%) 
coach. (n = 47) 
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Alignment 


TL21. Please indicate the degree to which you agree or disagree with the following 


statements. 
l l 
Strong y Disagree Agree Stronely 
disagree agree 
Our school connected LDC implementation to 2 (4.3%) | 17 (36.2%) | 17(36.2%) | 11 (23.4%) 
our specific schoolwide goals. (n = 47) 
LDC helped teachers create writing 
assignments to use within their current 1 (2.1%) 5 (10.6%) | 27(57.4%) | 14 (29.8%) 
curricula. (n = 47) 
LDC complemented other initiatives taking 1 (2.1%) | 10 (21.3%) | 25(53.2%) | 11 (23.4%) 
place in my school. (n = 47) 
| view LDC as a strategy for implementing my 
state’s College- and Career-Ready Standards. 1 (2.1%) 0 (0.0%) 31 (66.0%) | 15 (31.9%) 
(n = 47) 
The time spent implementing LDC interfered 
with other important initiatives at my school. | 5 (10.6%) | 18 (38.3%) | 20 (42.6%) 4 (8.5%) 
(n = 47) 
LDC helped prepare students in my school for 0 (0.0%) 5 (10.6%) | 28 (59.6%) | 14 (29.8%) 
current state assessments. (n = 47) 
It was difficult for teachers to focus on LDC 
because of other competing priorities at the 1 (2.1%) 8 (17.0%) | 24(51.1%) | 14 (29.8%) 
school. (n = 47) 
Our instructional leaders are using LDC to 
implement standards-driven assignments 1(2.1%) | 16(34.0%) | 20 (42.6%) | 10 (21.3%) 


within the existing curriculum. (n = 47) 
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Scale-up and Sustainability 


TL22. Please indicate the degree to which you agree or disagree with the following 


statements. 
Stroualy Disagree Agree Stronely 
disagree agree 
| expect that most teachers participating in 
LDC this year will continue to do so next year. 2 (4.3%) 12 (25.5%) | 22 (46.8%) | 11 (23.4%) 


(n =47) 


Teachers at my school who were not part of 
the LDC PLC meetings used the LDC planning 
process and/or LDC CoreTools. (n = 47) 


15 (31.9%) 


21 (44.7%) 


10 (21.3%) 


1 (2.1%) 


As a result of LDC, new collaborations across 
grades and/or subjects were created or are 
being launched at my school. (n = 47) 


4 (8.5%) 


19 (40.4%) 


20 (42.6%) 


4 (8.5%) 


Teachers and administrators at my school are 
committed to sustaining the LDC initiative. 
(n = 47) 


2 (4.3%) 


12 (25.5%) 


24 (51.1%) 


9 (19.1%) 


| expect our LDC PLC to increase in size next 
year. (n = 47) 


3 (6.4%) 


19 (40.4%) 


21 (44.7%) 


4 (8.5%) 
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(M0) District Support 


TL23. Please indicate the degree to which you agree or disagree with the following 
statements. 


Strongly Gleqeree ere Strongly Don’t 
disagree agree know 


District leaders supported the 


0 0, 0 0, 0 
nplementeeanot BCiNaee 0(0.0%) | 4(8.5%) | 17(36.2%) | 8(17.0%) | 18 (38.3%) 


District leaders had a firm 


0 0, 0 0, 0 
aceeraneerioc acka 1(2.1%) | 5(10.6%) | 7(14.9%) | 9(19.1%) | 25 (53.2%) 


District leaders are interested in 
spreading the use of LDC to 0 (0.0%) 4 (8.5%) 8 (17.0%) | 8(17.0%) | 27 (57.4%) 
additional schools. (n = 47) 


District professional 
development efforts were 
aligned with the LDC initiative. 
(n = 47) 


2 (4.3%) | 11(23.4%) | 13 (27.7%) | 7 (14.9%) | 14 (29.8%) 


District leaders visited my 
school to discuss the 3 (6.4%) 15 (31.9%) | 11(23.4%) | 6(12.8%) | 12 (25.5%) 
implementation of LDC. (n = 47) 


1y2) Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 
e CoreTools online platform 
e LDConline courses in the “Learn” section of CoreTools 
e Virtual coaching 
O Zoom meetings, written feedback on teacher work in LDC CoreTools, emails, etc. 
e  In-person coaching 


O Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 
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TL24. What supports did you find the most useful and why? 


TL25. What supports were not helpful and why? 


TL26. In what ways could LDC implementation be improved in your school in the future? 
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Appendix C: 
Administrator Survey and Responses 


2017-2018 
LDC School Administrator Survey 


i) LDC Participation 


A1. What is your role at the school? 


(n = 34) 
Principal 24 admins (70.6%) 
Assistant Principal 6 admins (17.6%) 
Other (please specify) ae 4 admins (11.8%) 


Professional Learning Community 


A2. What proportion of LDC Professional Learning Community (PLC) meetings did you 
attend during the current school year? 


(n = 34) 
Less than one quarter of LDC PLCs 7 admins (20.6%) 
About one quarter of LDC PLCs 6 admins (17.6%) 
About one half of LDC PLCs 9 admins (26.5%) 
About three quarters of LDC PLCs 3 admins (8.8%) 
More than three quarters of LDC PLCs 9 admins (26.5%) 
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Training and Support 


A3. How many in-person and/or online LDC professional development offerings for school 
administrators and teacher leaders did you attend during the current school year (e.g., 
Summer launch meetings, quarterly in-person administrator meetings, monthly virtual 
LDC leadership meetings)? 


| | LDC Professional development offerings n= 34, Mean = 5.26, Range: 0-20 


A4. Please indicate the degree to which you agree or disagree with the following statements. 


SiOnely Disagree Agree Strongly 
disagree agree 


N/A 


| was able to reach LDC staff when | 


0, 0, 0 Le) 0 
Gas enc unio: Geet 1 (2.9%) | 0(0.0%) | 10(29.4%) | 22 (64.7%) | 1(2.9%) 


My school has adequate 
technology to access LDC online 0(0.0%) | 0(0.0%) | 11(32.4%) | 23 (67.6%) | 0 (0.0%) 
resources. (n = 34) 


LDC offered sufficient professional 
development opportunities for LDC | 0(0.0%) | 1(2.9%) | 18(52.9%) | 15(44.1%) | 0 (0.0%) 
teacher leaders. (n = 34) 


LDC offered sufficient professional 
development opportunities for 0(0.0%) | 2(5.9%) | 17(50.0%) | 15(44.1%) | 0 (0.0%) 
school administrators. (n = 34) 


LDC staff members were able to 
connect me with additional 0(0.0%) | 0(0.0%) | 15 (44.1%) | 15 (44.1%) | 4(11.8%) 
resources when needed. (n = 34) 
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Classroom Observation 


A5. On average, how many times during the school year did you observe each member of 
the LDC PLC teaching an LDC module? 


(n = 34) 
0 times y Skip to A7 5 admins (14.7%) 
1 time 7 admins (20.6%) 
2 times 10 admins (29.4%) 
3 or more times 12 admins (35.3%) 


A6. On average, how effective were LDC modules in developing students’ literacy skills? 


(n = 29) 
Not effective 0 admins (0.0%) 
A little effective 4 admins (13.8%) 
Moderately effective 17 admins (58.6%) 
Very effective 8 admins (27.6%) 
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Impact on Teacher Practice 


A7. Based on your oversight of the LDC program, please indicate on average how much the 
teaching practice of LDC PLC members improved in each of the following areas: 


Not at all A little Moderately | A great deal 
Selecting a set of focus standards for a 0 (0.0%) 2 (5.9%) 16 (47.1%) 16 (47.1%) 
writing assignment (n= 34) 
Creating standards-driven writing 0 (0.0%) 7 (20.6%) 12 (35.3%) 15 (44.1%) 


assignments (n = 34) 


Identifying the skills students need to 
develop to complete a writing 0 (0.0%) 6 (17.6%) 17 (50.0%) 11 (32.4%) 
assignment (n = 34) 


Creating daily lessons to teach each 
skill a student needs to complete a 0 (0.0%) 7 (20.6%) 22 (64.7%) 5 (14.7%) 
writing assignment (n= 34) 


Systematically collecting information 


Le) 0, Le) ie) 
aie aes 2 (5.9%) 11 (32.4%) | 16(47.1%) | 5 (14.7%) 


Identifying patterns of student 
understandings or misconceptions 2 (5.9%) 10 (29.4%) 18 (52.9%) 4 (11.8%) 
(n = 34) 


Using evidence of student progress on 
standards to modify subsequent 2 (5.9%) 8 (23.5%) 17 (50.0%) 7 (20.6%) 
instruction (n = 34) 
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Impact on Student Learning 


A8. Please indicate to what extent LDC had a positive effect on students in the following 


areas: 
Not at all A little Moderately | A great deal 

Reading skills (n = 34) 1(2.9%) | 10(29.4%) | 17 (50.0%) 6 (17.6%) 
Content knowledge (n = 34) 0 (0.0%) 7 (20.6%) 18 (52.9%) 9 (26.5%) 
ca. complete writing assignments 0 (0.0%) 6 (17.6%) 14 (41.2%) 14 (41.2%) 
Quality of students’ writing (n = 34) 0 (0.0%) 7 (20.6%) 15 (44.1%) 12 (35.3%) 
College and career ready skills (n = 34) 0 (0.0%) 10 (29.4%) 18 (52.9%) 6 (17.6%) 
Capacity to analyze and understand the 
components of a writing assignment 0 (0.0%) 7 (20.6%) 20 (58.8%) 7 (20.6%) 
(n = 34) 
Speaking and listening skills (n = 34) 0 (0.0%) 10 (29.4%) 18 (52.9%) 6 (17.6%) 
Overall literacy performance (n = 34) 0 (0.0%) 7 (20.6%) 20 (58.8%) 7 (20.6%) 
Perf t 

LeinunetaaL cnet ani 0 (0.0%) | 13(38.2%) | 15 (44.1%) | 6 (17.6%) 


throughout the school year (n = 34) 
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¥/) Administrator Leadership Role 


AY. Please indicate the degree to which you agree or disagree with the following 
statements about your role in leading LDC implementation in your school: 


I I 
Stone v Disagree Agree stronely 
disagree agree 


| was able to shape LDC implementation at my 


0, 0, 0, 0 
sf at Mees: 0(0.0%) | 6(17.6%) | 26(76.5%) | 2 (5.9%) 


| met regularly with the LDC teacher leader in 
my school to stay abreast of implementation 0 (0.0%) 9 (26.5%) | 20(58.8%) | 5 (14.7%) 
progress. (n = 34) 


| was involved in discussions about 
differentiating LDC implementation to meet 0(0.0%) | 11 (32.4%) | 18(52.9%) | 5 (14.7%) 
teacher learning needs. (n = 34) 


| led discussions about how to expand my 
school’s LDC implementation in future years. 0(0.0%) | 10 (29.4%) | 19(55.9%) | 5 (14.7%) 
(n = 34) 


| made changes to school schedules to 
accommodate LDC professional learning time. 0 (0.0%) 4(11.8%) | 15 (44.1%) | 15 (44.1%) 
(n = 34) 


| allocated resources such as teacher time, 
payment, administrator time, support staff, 
sub coverage, etc., to ensure the LDC team 
could meet. (n = 34) 


0(0.0%) | 3(8.8%) | 17(50.0%) | 14 (41.2%) 
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Alignment 


A10. Please indicate the degree to which you agree or disagree with the following 


statements: 
Suenely Disagree Agree Suenely 
disagree agree 

Our school connected LDC implementation to 0 (0.0%) 5 (14.7%) | 25 (73.5%) | 4 (11.8%) 
our specific schoolwide goals. (n = 34) 
LDC helped teachers create writing 
assignments to use within their current 0 (0.0%) 0 (0.0%) 26 (76.5%) | 8 (23.5%) 
curricula. (n = 34) 
LDC complemented other initiatives taking 0 (0.0%) 0 (0.0%) 27 (79.4%) | 7 (20.6%) 
place in my school. (n = 34) 
| view LDC as a strategy for implementing my 
state’s College- and Career-Ready Standards. 0 (0.0%) 3 (8.8%) 24 (70.6%) | 7 (20.6%) 
(n = 34) 
The time spent implementing LDC interfered 
with other important initiatives at my school. 6 (17.6%) | 17 (50.0%) | 9 (26.5%) 2 (5.9%) 
(n = 34) 
LDC helped prepare students in my school for ‘ ‘ ‘ ° 
current state assessments. (n = 34) Oe) PIS): | COO! | SMe 
It was difficult for teachers to focus on LDC 
because of other competing priorities at the 1 (2.9%) 16 (47.1%) | 15 (44.1%) 2 (5.9%) 
school. (n = 34) 
| am using LDC to implement standards- 
driven assignments within the existin 0 (0.0% 2 (5.9% 27 (79.4% 5 (14.7% 

g g (0.0%) (5.9%) ( ) ( ) 
curriculum. (n = 34) 
LDC helped improve teacher evaluation 2 (5.9%) 16 (47.1%) | 13 (38.2%) 3 (8.8%) 


ratings. (n = 34) 
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Scale-up and Sustainability 


A11. Please indicate the degree to which you agree or disagree with the following 


statements: 


Strongly 
disagree 


Disagree 


Agree 


Strongly 
agree 


Don’t 
know 


| expect that most teachers 
participating in LDC this year will 
continue to do so next year. 

(n = 34) 


0 (0.0%) 


0 (0.0%) 


18 (52.9%) 


16 (47.1%) 


0 (0.0%) 


Teachers at my school who 
were not part of the LDC PLC 
meetings used the LDC planning 
process and/or LDC CoreTools. 
(n = 34) 


2 (5.9%) 


25 (73.5%) 


4 (11.8%) 


2 (5.9%) 


1 (2.9%) 


As a result of LDC, new 
collaborations across grades 
and/or subjects were created or 
are being launched at my 
school. (n = 34) 


0 (0.0%) 


11 (32.4%) 


18 (52.9%) 


5 (14.7%) 


0 (0.0%) 


Teachers and administrators at 
my school are committed to 
sustaining the LDC initiative. 

(n = 34) 


1 (2.9%) 


1 (2.9%) 


20 (58.8%) 


11 (32.4%) 


1 (2.9%) 


| expect our LDC PLC to increase 
in size next year. (n = 34) 


0 (0.0%) 


13 (38.2%) 


15 (44.1%) 


5 (14.7%) 


1 (2.9%) 
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District Support 


A12. Please indicate the degree to which you agree or disagree with the following 


statements: 
Strongly Dieseree ronae Strongly Don’t 
disagree agree know 
District leaders supported the 
0 (0.09 14.79 1 20 4 (11.89 7 (20.69 
implementation of LDC. (n = 34) OWA) | a me aa aes) ene) 
District leaders had a firm 
fo} fo} fo} fo} fo} 
iuderciandingof ibe eae 1 (2.9%) | 9(26.5%) | 11(32.4%) | 2(5.9%) | 11 (32.4%) 
District leaders are interested in 
spreading the use of LDC to 0 (0.0%) 3 (8.8%) 11 (32.4%) 2 (5.9%) | 18 (52.9%) 
additional schools. (n= 34) 
District professional development 
efforts were aligned with the LDC 1(2.9%) | 10(29.4%) | 13 (38.2%) | 4(11.8%) | 6 (17.6%) 
initiative. (n = 34) 
District leaders visited my school 
to discuss the implementation of 2 (5.9%) | 14(41.2%) | 14 (41.2%) | 2 (5.9%) 2 (5.9%) 


LDC. (n = 34) 
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1 1 Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 
e CoreTools online platform 
e LDConline courses in the “Learn” section of CoreTools 
e Virtual coaching 
O Zoom meetings, written feedback on teacher work in LDC CoreTools, emails, etc. 
e  /n-person coaching 


O Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 


A13. What supports did you find the most useful and why? 


A14. What supports were not helpful and why? 


A15. In what ways could LDC implementation be improved in your school in the future? 
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Appendix D: 
LDC Module Rating Dimensions 


Each module was rated on six dimensions. All of these were rated using a 5-point scale. With the first 
five, anchor points ranged from not present or realized to fully present or realized. In contrast, the 
overall dimension ranged from inadequate to advanced. 


Dimension 


1 Effective Writing Task 


Definition 

Degree to which teaching task makes effective use of the template task’s writing mode (i.e., argumentation or 
explanation); requires sustained writing and effective use of ideas and evidence to substantiate claims; and is 
feasible for most students to complete (i.e., appropriate for the grade-level and subject matter). 


Main Sources of Information 


Module Creator Handout (Task) 


e Read and evaluate the teaching task, student background/prior knowledge, and summary information. 
e Evaluate the difficulty or ease students may encounter trying to answer the question. 


e Compare module teaching task to teaching task template options. 


Anchor Points Description 


5 Fully Present or Realized The teaching task and performance expectations for the module are 
explicit and clear, require students to engage in higher-order thinking 
and writing, and are appropriate for the grade-level and subject matter. 


4  Sufficiently Present or Realized 


3 Moderately Present or Realized Clear module teaching task and performance expectations are 
available, but do not require students to engage in higher-order 
thinking and writing and/or are not appropriate for the grade-level and 
subject matter. 


2 Barely Present or Realized 


1 Not Present or Realized There is minimal evidence of an effort to identify an explicit and clear 
teaching task and performance expectations that provide opportunities 
for students to think critically and are appropriate for the grade-level 
and subject matter. 
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Dimension 


2 Standards 


Definition 


Alignment to CCSS, Local, State Literacy and Content 


Extent to which module addresses content essential to the discipline, as well as reading comprehension and 
writing standards informed by local and state standards. 


Main Sources of Information 


Module Creator Handout (Task) 


e Read and evaluate the standards included in the module. 


e Module should include ELA as well as subject matter CCSS/state standards. 


e Compare and contrast the standards the module includes with those that could have been included. 


e Particular attention to content standards (CCSS History/Social Studies, Science, and Technical Subjects); 
State Standards; Specific Reading, Writing, Speaking/Listening, Language Skills 


Anchor Points 


Description 


5 Fully Present or Realized 


Module specifically addresses content essential to CCSS and local or 
state standards in science or social studies, as well as reading 
comprehension and writing. All standards are well aligned to the topic 
and teaching task. 


4  Sufficiently Present or Realized 


3 Moderately Present or Realized 


Module broadly addresses content essential to CCSS and local or state 
standards in science or social studies and reading comprehension and 
writing. Standards are sufficiently aligned to the topic and teaching 
task. 


2 Barely Present or Realized 


1 Not Present or Realized 


Minimal evidence that module addresses content essential to the 
discipline and literacy standards. Standards are poorly aligned to the 
topic and teaching task. 
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Dimension 


3 Fidelity to LDC Module Instruction 


Definition 


Degree to which module instruction, activities, and teaching task address each of the four stages of instructional 
practice (preparation for the task, reading process, transition to writing, writing process). 


Main Sources of Information 


Module Creator Handout (Instruction) and Information Sheet 


e Evaluate for distribution of activities and time spent on each of the four stages of instructional practice. 


Anchor Points 


Description 


5 Fully Present or Realized 


The module instruction, activities, and teaching task reflect deliberate 
attention and fidelity to the four discrete stages of LDC module 
instruction. Classroom materials reflect demonstrable effort to develop 
instructional scaffolding within and across each stage of instruction. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


The module instruction, activities, and teaching task reflect moderate 
attention and fidelity to the four discrete stages of LDC module 
instruction. Classroom materials reflect sufficient effort to develop 
instructional scaffolding within and across each stage of instruction. 


2 Barely Present or Realized 


1 Not Present or Realized 


The module instruction, activities, and teaching task reflect poor 
attention and lack of fidelity to the four discrete stages of LDC module 
instruction. Classroom materials reflect inadequate effort to develop 
instructional scaffolding within and across each stage of instruction. 
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Dimension 


4 Quality Instructional Strategies 


Definition 

Degree to which the module provides clear instructional strategies aimed at helping students develop literacy skills 
and successfully complete the teaching task. In addition, the degree to which module instruction and activities 
scaffold critical thinking and performance in a way that is meaningful within the context of a given field or subject 
matter. 


Main Sources of Information 


Module Creator Handout (Instruction), Classroom Handouts, and Student Work 


e Evaluate extent to which the module activities scaffold critical thinking and student performance within 
the context of the subject matter at the core of the teaching task. 


e Evaluate extent to which instructional strategies guide student learning in literacy and ability to complete 
the teaching task. 


Anchor Points Description 


5 Fully Present or Realized Module provides clear and targeted instructional strategies and 
activities that scaffold student learning and promote critical thinking in 
social studies or science. There is explicit attention to helping students 
develop an accurate understanding of the topic and teaching task, and 
literacy skills necessary to complete the writing task successfully. 


4  Sufficiently Present or Realized 


3 Moderately Present or Realized Instructional strategies and activities are available to support adequate 
student learning and critical thinking in social studies or science. There 
is moderate attention to helping students develop an understanding of 
the topic and teaching task, and literacy skills necessary to complete 
the writing task. 


2 Barely Present or Realized 


1 Not Present or Realized Limited instructional strategies and activities are available to support 
student learning and critical thinking in social studies or science. 
Insufficient attention to helping students develop an understanding of 
the topic and teaching task, or literacy skills necessary to complete the 
writing task. 
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Dimension 


5 Coherence and Clarity of Module 


Definition 
The degree of logical alignment found between the teaching task and the goals of the module with the readings, 
mini-tasks, and instructional strategies. 


Main Sources of Information 


Module Creator Handout (Instruction), Classroom Handouts, and Student Work 


Anchor Points Description 


5 Fully Present or Realized Strong alignment between the teaching task and goals of the module— 
including the CCSS and local and state literacy and content standards— 
with the readings, mini-tasks, student work, and instructional 
strategies. 


4  Sufficiently Present or Realized 


3 Moderately Present or Realized Moderate alignment between the teaching task and goals of the 
module—including the CCSS and local and state literacy and content 
standards—with the readings, mini-tasks, student work, and 
instructional strategies. 


2 Barely Present or Realized 


1. Not Present or Realized Poor alignment between the teaching task and goals of the module— 
including the CCSS and local and state literacy and content standards— 
with the readings, mini-tasks, student work, and instructional 
strategies. 


138 


Dimension 


6 Overall Impression 


Definition 


A holistic assessment of the LDC Module. 


Main Sources of Information 


Module Creator Handout, Classroom Handouts, and Student Work 


e To what extent does this module contribute to student college readiness and development of advanced 
literacy skills? 


Anchor Points 


Advanced LDC Module Implementation 


Proficient LDC Module Implementation 


5 
4 
3 Adequate LDC Module Implementation 
2 


Marginal LDC Module Implementation 


1 Inadequate LDC Module Implementation 
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Appendix E: 
Research Procedure and Results for Principal Interviews 


To better understand teacher retention rates of LDC schools after the 2016-2017 school 
year, we conducted an interview study of school administrators. Twenty principals and assistant 
principals volunteered to be interviewed. This section presents the methods and results of the 
principal interviews. 


Method 


Instrument. We developed a six-question interview protocol for principals and assistant 
principals. Some of the interview questions had subquestions. Five of the questions were asked 
of all principals, regardless of their schools’ retention typology, and one question (Question 5) 
differed slightly based on the school’s retention typology. See Figure E1 for the interview 
protocol. 


140 


Question no. Question Given to 


1 Why did you initially participate in LDC? All 
e What were your goals when you signed on and how well have those 
goals been met thus far? 
2 What was your role in LDC implementation? All 
e Who else at the school supported LDC? 
e How much did you see of PLC meetings? 
e Was part of your role to give feedback on LDC instruction? (If 
needed: Who gave feedback on LDC instruction?) 


3 Could you tell me about outside support — from both LDC and the All 
district — and what worked or didn’t work? 


4 What were the incentives for teachers’ participation in LDC? All 
e How would you describe teacher buy-in and commitment to LDC? 


e Did teachers receive any additional pay? If so, for what? 


5 What would you say are the main factors that led to not continuing with Drop out 
LDC? 


a) Looking across the district, we noticed that some schools chose to Zero 
continue with LDC for another year and others chose to stop. What retention 
would you say are the main factors that led to continuing with LDC? 


b) We noticed that none of the year-1 teachers participated in Year 2, 
so the year-2 PLC began with an entirely new set of teachers. Why 
do you think that was? 


c) Were there changes in implementation to keep this from 
happening again the following year? 


a) Looking across the district, we noticed that some schools chose to Very Low 
continue with LDC for another year and others chose to stop. What and Low 
would you say are the main factors that led to continuing with LDC? Retention 


b) We noticed that many teachers who participated in Year 1 did not 
participate in Year 2. Why do you think that was? 


a) Looking across the district, we noticed that some schools chose to Moderate- 
continue with LDC for another year and others chose to stop. What High 
would you say are the main factors that led to continuing with LDC? Retention 


b) We noticed that many teachers who participated in Year 1 
continued into Year 2. Why do you think that was? 


6 Do you have any advice for strengthening LDC or is there anything else A 
we should know about your experience with LDC? 


Figure E1. Principal and assistant principal interview protocol. 


Participants. During the 2016-2017 school year, 48 total schools participated in the i3 LDC 
program including 28 in the New York City Department of Education (NYCDOE) and 20 in the 
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West Coast district which is the subject of this report. We categorized the schools into retention 
typologies based on a school’s total teacher participants in 2016-2017 and returning teachers 
in 2017-2018. Using the proportions of the same teacher participating for the two years, we 
created five school retention typologies: 


e drop out (the school has no teachers participating in LDC in 2017-2018); 


e zero retention (no 2016-2017 teachers returning to LDC, but new teachers in the 
school are participating in 2017-2018); 


e very low retention (less than 1/3 of teachers returning); 

e lowretention (between 1/3 and 2/3 returning teachers); and 

e moderate to high retention (over 2/3 teachers returning). 

Table E1 shows the number of schools per retention typology in NYCDOE and the West 


Coast district. 


Table E1 


Number of Schools and Proportion of Retention Typologies for NYCDOE and West Coast district in 
2016-2017 


# West Coast % West Coast 


# NYCDOE % NYCDOE district district 
Retention type schools schools schools schools 
Dropped out after 2016-2017 10 36% 6 30% 
Zero retention 1 4% 2 10% 
Very low teacher retention 7 25% 3 15% 
Low teacher retention 5 18% 2 10% 
Moderate to high teacher retention 5 18% 7 35% 
Total 28 20 


All LDC schools from the 2016-2017 cohort were contacted for a phone interview by 
email, and a total of 20 principals and assistant principals (10, or 36%, of NYCDOE schools and 
10, or 50%, of West Coast district schools) participated in the interview. Table E2 shows the 
school retention typologies that are represented in this sample. Of the principals and assistant 
principals who participated in an interview, most were from very low teacher retention schools 
(eight participants), followed by moderate to high teacher retention schools (six participants). 
We also had three principals or assistant principals from drop out schools participate, two 
principals from low teacher retention schools, and one principal in a zero retention school. 
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Table E2 
School Retention Typologies Represented in the Study 


# West Coast % West Coast 


# NYCDOE % NYCDOE district district 
Retention Type schools schools schools schools 
Dropped out after 2016-2017 2 20% 1 17% 
Zero retention 0 0% 1 50% 
Very low teacher retention 5 71% 3 100% 
Low teacher retention 1 20% 1 50% 
Moderate to high teacher retention 2 40% 4 57% 
Total 10 10 


Procedures. We originally contacted principals of all schools except for schools designated 
as low retention. This was done as sampling strategy for a case study methodology to 
understand reasons behind high and low retention schools. Initial emails were sent to 41 
schools in both districts (23 in NYCDOE and 18 in the West Coast district) the week of June 16, 
2018. A minimum of three follow-up emails for nonresponding administrators were sent 
between June 25 and August 6, 2018. During this window, nine NYCDOE principals or assistant 
principals participated (39% of schools in the first round) and seven principals from the West 
Coast district participated (39% of schools in the first round). One West Coast district principal 
contacted us to say that she did not have enough knowledge about the LDC program in the 
school because she was the new principal. 


Because of the low response rate, we decided to contact the principals in the low teacher 
retention schools (five in NYCDOE and two in the West Coast district) to increase the numbers 
of participants. The initial emails for these NYCDOE principals were sent on September 20, 
2018, and for West Coast district principals, between October 1 and October 8, 2018. Follow-up 
emails were sent to NYCDOE and West Coast district principals between the weeks of October 1 
and October 19, 2018. We also reached out to the LDC grant directors in NYCDOE and the West 
Coast district for recruitment support. Between the new round of emails and support from the 
LDC grant directors, four additional school leaders participated in interviews, bringing our total 
participants to 20 principals and assistant principals (10, or 36%, of NYCDOE schools and 10, or 
50%, of West Coast district schools). 


Interviews were scheduled at times most convenient to the principal or assistant 
principal. Interviews were conducted by two researchers following the protocol, and the 
interviews ranged from about eight minutes to 30 minutes in length. Participant consent was 
obtained prior to the interviews, and interviews were recorded. 
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Analysis. Recordings of interviews were transcribed. (For one interview, due to researcher 
and equipment error, the interview was not recorded. However, notes taken by the researcher 
during the interview were used for general counts in the analysis.) Transcriptions were read 
iteratively and coded for broad themes using grounded theory (Glaser & Strauss, 1967). 
Multiple close readings of each group transcript were conducted in order to summarize the 
data and identify themes related to reasons for a school’s retention (or lack thereof) of teachers 
from Year 1 to Year 2 of LDC implementation. A coding scheme was developed based on 
targeted areas of interest in relation to reasons for retention. 


Results 


Qualitative results from the data are presented in five main categories: reasons for 
teachers continuing in LDC, meeting goals, administrator’s role, teacher buy-in, and outside 
support. 


Reasons for teachers continuing or leaving LDC in Year 2. To understand why teachers 
remained or left LDC after Year 1, interview data were coded into three main categories: 
teacher or grade-level decision, administrative decision, and mandatory participation. In 
teacher or grade-level decisions, individual teachers, a grade level, or a content-area team were 
free to decide to return the following year. Administrative decisions typically involved the 
principal deciding on changing the grade level or content-area teachers who would participate 
in the second year of implementation (e.g., moving from second-grade to third-grade teachers; 
dropping mathematics teachers but keeping science teachers in the program). A couple of 
principals also decided that LDC participation would be better suited for a particular level of 
teacher. For example, a principal decided to focus Year 2 teacher participation on “teachers 
that were new to the profession and teachers that were struggling instructionally”; therefore, 
only a few teachers continued from Year 1 to Year 2 at that site. Mandatory participation, 
which is based on an administrator’s decision, reflected schools where teachers were told that 
participation in the second year of LDC was not voluntary. Table E3 shows the reasons for 
retention by school retention typology. 
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Table E3 
Teachers’ Participation in LDC in Year 2 by Retention Typology 


Teacher/grade Administrative Mandatory 
Retention typology level decision decision participation 
Dropped out after 2016-2017 1 2 0 
Zero retention 0 1 
Very low teacher retention 4 4 0 
Low teacher retention 1 0 1 
Moderate to high teacher retention 4 0 2 
Total 10 7 3 


For the three drop out schools in our sample, two of the schools had a change in 
leadership. One had a new principal who came in before the second year of LDC 
implementation and decided not to participate in LDC, and the other school had a new 
superintendent who did not continue partnering with LDC. For the last of the three drop out 
schools, the small group of teachers who were in the first year of implementation decided not 
to continue because of “teacher and administrator burn out.” This school had too many 
initiatives that were handled by a small number of administrative staff and the same teachers 
who were also part of the LDC implementation. 


For the one zero retention school in our sample, the principal decided to change the 
teachers for Year 2. The grade level that was selected to participate in LDC the first year found 
the program successful, and so for Year 2, “other teachers haven't been exposed, and we 
wanted more teachers to have access ... to spread the wealth within the school,” so another 
grade level was selected to participate. In one of the low retention schools, the principal said 
that participation in LDC was mandatory, but many of the teachers who were part of Year 1 
implementation left the school the following year. 


For the 10 schools that were coded as a teacher/grade-level decision, half of those 
decisions were based on the grade-level or content-area team (e.g., science teachers) deciding 
to leave LDC because they started a new program or initiative the second year and did not have 
the time to participate in LDC. For the moderate to high teacher retention schools who were 
coded as teacher/grade-level decision, the majority of teachers in those schools decided to stay 
in LDC the second year. 


Meeting goals. We asked participants their reasons for participating in LDC during the 
first year. The most popular reasons for participating in LDC were its alignment to standards 
and rigorous tasks. Principals and assistant principals also stated that they felt both teachers 
and students in their schools needed assignments where literacy instruction, the writing tasks, 
and the writing process were a focus. Other responses included wanting better lesson planning, 
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collaboration, and content-area instruction. Interestingly, three principals said that LDC 
appealed to them as they were new to the school or position. 


After stating their reasons for participation, we asked if these goals were met after the 
first year of implementation. Table E4 shows participant responses to whether goals were met 
by school typology. The majority of principals and assistant principals responded positively: 12 
(60%) said yes, five expressed mixed feelings about accomplishing their original goals for LDC 
(25%), and three said that their goals were not met (15%). From these responses, meeting goals 
for LDC did not appear to influence retention of teachers. For example, two of the three 
dropout schools reported meeting their LDC goals for participation, yet they didn’t remain in 
the program. Teacher retention seemed to be tied more to teacher and principal decisions (see 
section above). 


Table E4 
Principal Reponses for LDC Goals by School Typology 


LDC goals met 


School typology Yes No Mixed 
Dropped out after 2016-2017 2 1 0 
Zero retention 1 0 0 
Very low teacher retention 4 1 3 
Low teacher retention 1 1 0 
Moderate to high teacher retention 4 0 2 
Total 12 3 5 


Administrator's role. The majority of principals and assistant principals in our sample 
reported having an active role in supporting LDC implementation. Sixteen (80%) attended LDC 
PLCs, and those who could not make the PLCs were able to get a summary of the meetings from 
their assistant principals or lead teachers. For those who were able to attend some PLCs, the 
amount of time they were able to sit in on the PLCs varied. Some principals were only able to 
drop in for a portion of the PLCs. For example, one principal (moderate to high retention 
school) said, “| would probably just go in the last 15 minutes. | don't think | was in a full session 
all the time. But | would touch bases and then each teacher would give me a wrap up summary 
of what they had done, what they were working on." Others were able to go to almost all PLCs. 
As the zero retention school principal said, “| was a participant, also, in the learning.” Sixteen 
participants (80%) also reported that they gave feedback on LDC instruction. However, the 
degree to which principals or assistant principals gave feedback varied. Table E5 shows the 
numbers of participants who attended PLCs and provided feedback on LDC instruction. 
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Table E5 


Participation of Principals and Assistant Principals in LDC Implementation by 
Retention Type 


Retention typology Attended PLCs —_ Provided feedback 
Dropped out after 2016-2017 3 3 
Zero retention 1 1 
Very low teacher retention 6 6 
Low teacher retention 1 2 
Moderate to high teacher retention 5 4 


Total 16 16 


Teacher buy-in. Most of the principals and assistant principals mentioned increasing skills 
and knowledge, collaboration, and student achievement and accessing LDC materials as 
incentives for teachers. Besides these incentives that were actually part of the LDC 
implementation, a few schools were able to provide additional school-based supports. Eight 
schools (40%) in our sample reported providing teachers with more tangible incentives for their 
participation in LCD, such as additional pay and time, and four of the eight gave multiple 
incentives. Table E6 shows the number of these types of teacher incentives by school typology. 
No pattern emerges that connects teacher incentives to retention typology. However, it is 
notable that the majority of schools in the sample did not provide tangible incentives (i.e., pay, 
extra time, teacher evaluation) to teachers for their participation for LDC. 


Table E6 
Teacher Incentives by School Typology 


Retention typology Additional pay Extratime Teacher evaluation 


Dropped out after 2016-2017 
Zero retention 

Very low teacher retention 
Low teacher retention 


Moderate to high teacher retention 


ulrP PN RF Oo 
BlP NM OF O 
Nn); rF oO F ODO O 


Total 


Five schools (25%) were able to pay teachers for their participation in LCD. Aside from 
pay, some principals were able to support participating teachers with extra time, such as 
adjusting the schedule to give teachers more time to meet during the week or giving teachers a 
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release day. For example, the principal from the zero retention school said, “Initially, it was kind 
of an imposition of time, but when we created extra time and space for the teachers to meet, 
they were appreciative. So they had more buy-in.” Another principal (low retention school) 
said, 

| make it very easy for teachers, because I've built in time in their 

schedule for them to meet for LDC. There are no discrepancies in terms 

of when we should meet or how can we meet, or what needs to be done, 

which is usually the issue sometimes when it comes to work with outside 

consultants or providers, right? 


Two schools incentivized participation through teacher evaluations. One principal (very 
low retention school) postponed teacher evaluations for the year. The other principal 
(moderate to high retention school) said, “I very much tied it to teacher performance and my 
expectation of what high quality planning looks like.” For this principal, he knew LDC would be 


transformative. | knew that if they got through it and actually did it, the 
carrot would reveal itself in the students’ performance. And | knew my 
teachers were committed; | knew they wanted our kids to do well. They 
just needed to be pushed through the process to see how to get the kids 
to do well. 


Outside support. Overwhelmingly positive responses were reported from our sample 
regarding LDC support the schools received. All principals and assistant principals mentioned 
either training or coaching that LDC provided. Two principals, both from very low retention 
schools, also mentioned that the administrator meetings were helpful. As one principal 
described, “It gave an opportunity to speak to other people who were in the project, and it also 
gave an opportunity to actually go through all of the CoreTools that were there to support the 
teachers.” Two principals (both from New York) also found site visits to another school helpful: 
“There are sites where the work is going on really nicely, and we were able to go over to those 
sites and see presentations from the principal and her teachers on how they have embedded 
this entire system into their daily program.” Five participants mentioned district support being 
available. 


Discussion 


Participants in this study all reported playing an active role in LDC implementation at their 
school site, and most said that their goals for participation were met. They also overwhelmingly 
responded favorably to LDC support, and district support, if provided or perceived, was also 
helpful. The principal interviews showed the diversity of reasons for why schools were able to 
retain teachers, and they also show why teachers left. The main reason for teacher retention 
was teacher and grade-level team decisions to leave or stay, followed by the principals’ decision 
to switch participation of teachers from Year 1 to Year 2. Teacher buy-in also seemed to affect 
teachers’ decisions for remaining or continuing. We did not find any consistent trends for 
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retention typology; that is, we cannot characterize reasons for each retention typology and say, 
for example, all drop out schools had these characteristics or most moderate to high retention 
schools had other characteristics. Teacher retention in LDC from Year 1 to Year 2 was due toa 
myriad of factors that worked in combination at each school. 
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Appendix F: 
Additional Results on Module Artifact Ratings 


The following presents further details of the module analyses. First we present the 
methodology and results of the generalizability theory studies. This is followed by the 
additional tables for the descriptive analyses. 


Generalizability Study 


Generalizability theory is a statistical framework for examining multiple sources of 
potential error during the rating process. For each grade band, we first modeled variability in 
ratings across all six dimensions using a two-faceted design, whereby we estimated variance 
components for module by rater by dimension (t x r x d). The goal here was to separate true 
variation in the modules from other potential sources of measurement error. The main effects 
reflect true variation across modules (o2t) and error variance across raters (o2r) and 
dimensions (02d), while the error term (o2trd,e) reflects unexplained residual error in the 
model. To disentangle the sources of potential error further, we also used a single-faceted 
design to examine potential error within the scoring of each dimension. As with the first set of 
models, the main effect reflects true variation across teachers (o2t) and error variance across 
raters (o2r). 


Elementary module results. Generalizability theory models were conducted to examine 
potential error in the scoring process for the elementary modules. Tables F1 and F2 present 
results from the two-faceted and one-faceted models that examine error across and within 
dimensions. As we expected, most of the variation found for the elementary modules was due 
directly to differences in the modules (46%) or to differences in the modules by dimension 
(31%). Despite this, 16% of the variation was due either directly or through interaction with the 
raters, and 6% of the variation was unexplained by the two-faceted model used. 
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Table F1 


Generalizability Study of the Elementary Module 


Ratings Across Dimensions (n = 140) 


Source Var. % 
Module (02t) 0.45 46.13 
Rater (o2r) 0.08 8.10 
Dimension (o2d) 0.01 1.32 
Module x Dimension (o2td) 0.30 30.73 
Rater x Dimension (o2rd) 0.06 6.02 
Module x Rater (o2tr) 0.02 1.87 
Error (o2trd,e) 0.06 5.84 


As previously mentioned, we also used a one-faceted design to disentangle variation in 
the ratings that was due either directly or through interaction with the dimensions (see Table 
F2). As would be desired, the greatest source of variation for each dimension was due directly 
to differences in the modules. Despite this, moderate amounts of variation could be attributed 
either directly or through interaction with the raters for Dimensions 3 through 6. For example, 
with Dimension 4, which measures the quality of instructional strategies, 28% of the variation 


was due to the raters and an additional 6% was due to an interaction between raters and 
modules. More than 25% of the variation in ratings for Dimension 6 was also attributable 


directly to the raters. 


Table F2 


Generalizability Study of the Elementary Module Ratings for Each Dimension (n = 140) 


Module x Rater Error 

Module (o2t) Rater (o2r) (o2tr) (o2trd,e) 

Dimension Var. % Var. % Var. % Var. % 
1. Effective writing task 0.50 87.80 0.03 5.38 0.04 6.82 0.00 0.00 
2. Standards alignment 0.97 92.12 0.08 7.88 0.00 0.00 0.00 0.00 
3. Fidelity to LDC instruction 0.75 73.72 0.04 3.49 0.23 22.79 0.00 0.00 
4. Quality instructional strategies 0.75 66.19 0.31 27.72 0.07 6.09 0.00 0.00 
5. Coherence/clarity of module 0.73 73.00 0.12 11.90 0.15 15.09 0.00 0.00 
6. Overall impression 0.62 57.98 0.27 25.29 0.18 16.73 0.00 0.00 
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Secondary module results. Generalizability theory models were also used to examine 
potential error in the scoring process for the secondary modules. Tables F3 and F4 present 
results from the two-faceted and one-faceted models that examine error across and within 
dimensions. As would be the goal of any rating session, most of the variation found in the 
ratings was due directly to differences in the modules (42%) or to differences in the modules by 
dimension (48%). Furthermore, less than 3% of the variation for the secondary modules was 
due either directly or indirectly to the raters. What is of concern, though, is that 7% of the 
variation in ratings for the two-faceted model was unexplained. 


Table F3 


Generalizability Study of the Secondary Module 
Ratings Across Dimensions (n = 86) 


Source Var. % 
Module (02t) 0.48 41.98 
Rater (o2r) 0.02 2.10 
Dimension (o2d) 0.00 0.00 


Module x Dimension (o2td) 0.55 47.69 


Rater x Dimension (o2rd) 0.01 0.82 
Module x Rater (o2tr) 0.00 0.00 
Error (o2trd,e) 0.09 7.41 


We also used a one-faceted design to disentangle the variance that was due either 
directly or through interaction with the dimensions (see Table F4). As would be desired, more 
than three quarters of the variance found for each dimension was attributable to differences in 
the modules. In general, only zero to 5% of the variance was due directly to the raters. The only 
exception involved coherence and clarity of the module (Dimension 5), in which 11% of the 
variation in ratings was due to the raters. Results were similar regarding the interaction 
between modules and raters, with very little variance found for the dimensions, except the first 
one that focuses on the effective writing task (23%). Finally, there was no unexplained error 
variance found for any of the one-faceted models. 
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Table F4 
Generalizability Study of the Secondary Module Ratings for Each Dimension (n = 86) 


Module x Rater Error 

Module (o2t) Rater (o2r) (o2tr) (o2trd,e) 

Dimension Var. % Var. % Var. % Var. % 
1. Effective writing task 1.24 76.73 0.00 0.00 0.38 23.27. 0.00 0.00 
2. Standards alignment 1.06 97.29 0.03 2.71 0.00 0.00 0.00 0.00 
3. Fidelity to LDC instruction 1.09 99.84 0.00 0.00 0.00 0.16 0.00 0.00 
4. Quality instructional strategies 0.99 91.85 0.05 4.99 0.03 3.16 0.00 0.00 
5. Coherence/clarity of module 1.02 88.34 0.12 10.74 0.01 0.92 0.00 0.00 
6. Overall impression 0.88 95.06 0.05 4.94 0.00 0.00 0.00 0.00 


Note. Negative estimates of variance were changed to zero in order to calculate percentages (see Shavelson & 
Webb, 1991). 


Summary. As previously noted, generalizability models were fit for the overall samples for 
the two grade bands. As would be hoped for, when examining the one-faceted models, the 
majority of variation for the elementary and secondary ratings were due directly to differences 
in the modules (46%, 42%) or to differing quality in the modules across dimensions (31%, 48%). 
Likewise, results from the two-faceted models showed that the majority of variation within 
dimension was due to differences in the models (58% to 92%). Despite this, about one quarter 
of the variation for Dimension 4, which measures quality instructional strategies, and 
Dimension 6, which measures overall impression, was due directly to the raters. Similarly, 
almost one quarter of the variation in ratings for Dimension 3, which measures fidelity to LDC 
instruction, was due to an interaction between raters and modules. 


Descriptive Results 


The following section presents expanded descriptive results for both the primary and 
secondary modules. We first present background information about the modules rated. This is 
followed by descriptive statistics and percentages for the elementary modules and the 
secondary modules. Finally, we present further descriptive statistics and percentages for the 
exploratory analysis of elementary modules. 
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Table F5 
West Coast District Modules—Background Variables for the Primary Module Analysis 


Elementary Secondary Total 
Variables # % # % # % 

Cohort 

Cohort 1 returning 12 11.32 15 23.44 27 15.88 

Cohort 1 new 11 10.38 7 10.94 18 10.59 

Cohort 2 83 78.30 42 65.63 125 73.53 
Subject 

ELA 43 40.57 28 43.75 71 41.76 

Science 43 40.57 16 25.00 59 34.71 

Social Studies 20 18.87 20 31.25 40 23.53 
Module origin 

Adapted (Other authors) 68 64.15 38 59.38 106 62.35 

Adapted (Same author) 31 29.25 20 31.25 51 30.00 

Original 7 6.60 6 9.38 13 7.65 
Author count 

Coauthored 44 41.51 20 31.25 64 37.65 

Sole 62 58.49 44 68.75 106 62.35 
Module type 

One week, one text 6 5.66 2 3.13 8 4.71 

Regular 100 94.34 62 96.88 162 95.29 
Module components 

Student background 89 83.96 46 71.88 135 79.41 

Extension 45 42.45 18 28.13 63 37.06 

Teacher reflection 31 29.25 23 35.94 54 31.76 
Total 106 62.35 64 37.65 170 100.00 


Note. Teacher reflection counts calculated from the metadata. 
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Table F6 


Descriptive Statistics for the Elementary Modules by Content Area 


Dimension M SD Median Mode Minimum Maximum 

ELA (n = 43) 

1. Effective writing task 4.19 0.79 4.00 5.00 3.00 5.00 
2. Standards alignment 4.16 0.69 4.00 4.00 3.00 5.00 
3. Fidelity to LDC instruction 3.86 1.23 4.00 4.00 1.00 5.00 
4. Quality instructional strategies 3.86 1.01 4.00 4.00 1.00 5.00 
5. Coherence/clarity of module 3.93 1.03 4.00 4.00 1.00 5.00 
6. Overall impression 3.77 1.13 4.00 4.00 0.00 5.00 


Science (n = 43) 


1. Effective writing task 4.54 0.74 5.00 5.00 2.00 5.00 
2. Standards alignment 3.93 1.12 4.00 5.00 2.00 5.00 
3. Fidelity to LDC instruction 4.47 0.91 5.00 5.00 1.00 5.00 
4. Quality instructional strategies 4.56 0.88 5.00 5.00 1.00 5.00 
5. Coherence/clarity of module 4.02 1.12 4.00 5.00 1.00 5.00 
6. Overall impression 4.30 0.86 5.00 5.00 2.00 5.00 


Social studies (n = 20) 


1. Effective writing task 4.30 0.73 4.00 5.00 3.00 5.00 
2. Standards alignment 3.95 0.69 4.00 4.00 2.00 5.00 
3. Fidelity to LDC instruction 4.05 0.89 4.00 4.00 2.00 5.00 
4. Quality instructional strategies 3.60 1.10 4.00 4.00 1.00 5.00 
5. Coherence/clarity of module 3.70 0.80 4.00 4.00 2.00 5.00 
6. Overall impression 3.55 1.10 4.00 3.00 0.00 5.00 


Overall (n = 106) 


1. Effective writing task 4.35 0.77 5.00 5.00 2.00 5.00 
2. Standards alignment 4.03 0.89 4.00 4.00 2.00 5.00 
3. Fidelity to LDC instruction 4.14 1.07 4.00 5.00 1.00 5.00 
4. Quality instructional strategies 4.09 1.05 4.00 5.00 1.00 5.00 
5. Coherence/clarity of module 3.93 1.03 4.00 5.00 1.00 5.00 
6. Overall impression 3.94 1.06 4.00 5.00 0.00 5.00 
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Table F7 


Distribution (Percentage) of Ratings for the Elementary Modules by Content Area 


Dimension 1 2 3 4 5 
ELA (n = 43) 
1. Effective writing task 0.00 0.00 23.26 34.88 41.86 
2. Standards alignment 0.00 0.00 16.28 51.16 32.56 
3. Fidelity to LDC instruction 9.30 4.65 11.63 39.53 34.88 
4. Quality instructional strategies 2.33 6.98 23.26 37.21 30.23 
5. Coherence/clarity of module 2.33 6.98 20.93 34.88 34.88 
6. Overall impression 0.00 11.63 18.60 39.53 27.91 
Science (n = 43) 
1. Effective writing task 0.00 2.33 6.98 25.58 65.12 
2. Standards alignment 0.00 16.28 16.28 25.58 41.86 
3. Fidelity to LDC instruction 2.33 0.00 13.95 16.28 67.44 
4. Quality instructional strategies 2.33 2.33 4.65 18.60 72.09 
5. Coherence/clarity of module 2.33 9.30 18.60 23.26 46.51 
6. Overall impression 0.00 2.33 18.60 25.58 53.49 
Social studies (n = 20) 
1. Effective writing task 0.00 0.00 15.00 40.00 45.00 
2. Standards alignment 0.00 5.00 10.00 70.00 15.00 
3. Fidelity to LDC instruction 0.00 5.00 20.00 40.00 35.00 
4. Quality instructional strategies 5.00 10.00 25.00 40.00 20.00 
5. Coherence/clarity of module 0.00 5.00 35.00 45.00 15.00 
6. Overall impression 0.00 0.00 40.00 40.00 15.00 
Overall (n = 106) 
1. Effective writing task 0.00 0.94 15.09 32.08 51.89 
2. Standards alignment 0.00 7.55 15.09 44.34 33.02 
3. Fidelity to LDC instruction 4.72 2.83 14.15 30.19 48.11 
4. Quality instructional strategies 2.83 5.66 16.04 30.19 45.28 
5. Coherence/clarity of module 1.89 7.55 22.64 32.08 35.85 
6. Overall impression 0.00 5.66 22.64 33.96 35.85 
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Table F8 
Descriptive Statistics for the Elementary Modules by Cohort 


Dimension M SD Median Mode Minimum Maximum 


Cohort 1 returning (n = 12) 


1. Effective writing task 3.67 0.49 4.00 4.00 3.00 4.00 
2. Standards alignment 3.75 0.97 4.00 3.00 2.00 5.00 
3. Fidelity to LDC instruction 3.75 1.06 4.00 4.00 2.00 5.00 
4. Quality instructional strategies 3.25 1.14 3.50 4.00 1.00 5.00 
5. Coherence/clarity of module 3.33 1.07 3.00 3.00 2.00 5.00 
6. Overall impression 3.17 1.11 3.00 3.00 0.00 4.00 


Cohort 1 new (n = 11) 


1. Effective writing task 4.36 0.92 5.00 5.00 3.00 5.00 
2. Standards alignment 4.27 0.90 4.00 4.00 2.00 5.00 
3. Fidelity to LDC instruction 4.00 1.61 5.00 5.00 1.00 5.00 
4. Quality instructional strategies 4.18 1.33 5.00 5.00 1.00 5.00 
5. Coherence/clarity of module 4.09 1.38 5.00 5.00 1.00 5.00 
6. Overall impression 4.18 1.25 5.00 5.00 2.00 5.00 


Cohort 2 (n = 83) 


1. Effective writing task 4.45 0.74 5.00 5.00 2.00 5.00 
2. Standards alignment 4.04 0.88 4.00 4.00 2.00 5.00 
3. Fidelity to LDC instruction 4.22 0.99 4.00 5.00 1.00 5.00 
4. Quality instructional strategies 4.21 0.95 4.00 5.00 1.00 5.00 
5. Coherence/clarity of module 3.99 0.96 4.00 4.00 1.00 5.00 
6. Overall impression 4.02 0.99 4.00 5.00 0.00 5.00 


Overall (n = 106) 


1. Effective writing task 4.35 0.77 5.00 5.00 2.00 5.00 
2. Standards alignment 4.03 0.89 4.00 4.00 2.00 5.00 
3. Fidelity to LDC instruction 4.14 1.07 4.00 5.00 1.00 5.00 
4. Quality instructional strategies 4.09 1.05 4.00 5.00 1.00 5.00 
5. Coherence/clarity of module 3.93 1.03 4.00 5.00 1.00 5.00 
6. Overall impression 3.94 1.06 4.00 5.00 0.00 5.00 
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Table F9 


Distribution (Percentage) of Ratings for the Elementary Modules by Cohort 


Dimension 1 2 3 4 5 
Cohort 1 returning (n = 12) 
1. Effective writing task 0.00 0.00 33.33 66.67 0.00 
2. Standards alignment 0.00 8.33 33.33 33.33 25.00 
3. Fidelity to LDC instruction 0.00 16.67 16.67 41.67 25.00 
4. Quality instructional strategies 8.33 16.67 25.00 41.67 8.33 
5. Coherence/clarity of module 0.00 25.00 33.33 25.00 16.67 
6. Overall impression 0.00 0.00 50.00 41.67 0.00 
Cohort 1 new (n = 11) 
1. Effective writing task 0.00 0.00 27.27 9.09 63.64 
2. Standards alignment 0.00 9.09 0.00 45.45 45.45 
3. Fidelity to LDC instruction 18.18 0.00 9.09 9.09 63.64 
4. Quality instructional strategies 9.09 0.00 18.18 9.09 63.64 
5. Coherence/clarity of module 9.09 0.00 27.27 0.00 63.64 
6. Overall impression 0.00 18.18 9.09 9.09 63.64 
Cohort 2 (n = 83) 
1. Effective writing task 0.00 1.20 10.84 30.12 57.83 
2. Standards alignment 0.00 7.23 14.46 45.78 32.53 
3. Fidelity to LDC instruction 3.61 1.20 14.46 31.33 49.40 
4. Quality instructional strategies 1.20 4.82 14.46 31.33 48.19 
5. Coherence/clarity of module 1.20 6.02 20.48 37.35 34.94 
6. Overall impression 0.00 4.82 20.48 36.14 37.35 
Overall (n = 106) 
1. Effective writing task 0.00 0.94 15.09 32.08 51.89 
2. Standards alignment 0.00 7.55 15.09 44.34 33.02 
3. Fidelity to LDC instruction 4.72 2.83 14.15 30.19 48.11 
4. Quality instructional strategies 2.83 5.66 16.04 30.19 45.28 
5. Coherence/clarity of module 1.89 7.55 22.64 32.08 35.85 
6. Overall impression 0.00 5.66 22.64 33.96 35.85 
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Table F10 
Descriptive Statistics for the Secondary Modules by Content Area 


Dimension M SD Median Mode Minimum Maximum 

ELA (n = 28) 

1. Effective writing task 3.54 1.20 4.00 3.00 1.00 5.00 
2. Standards alignment 3.54 0.92 3.50 3.00 1.00 5.00 
3. Fidelity to LDC instruction 3.68 1.22 4.00 4.00 1.00 5.00 
4. Quality instructional strategies 4.00 1.15 4.00 5.00 1.00 5.00 
5. Coherence/clarity of module 4.11 0.92 4.00 4.00 1.00 5.00 
6. Overall impression 3.86 0.93 4.00 4.00 1.00 5.00 


Science (n = 16) 


1. Effective writing task 3.56 0.89 4.00 4.00 2.00 5.00 
2. Standards alignment 3.25 1.24 3.50 2.00 2.00 5.00 
3. Fidelity to LDC instruction 3.56 1.03 3.50 3.00 1.00 5.00 
4. Quality instructional strategies 3.38 1.09 3.50 4.00 1.00 5.00 
5. Coherence/clarity of module 3.13 1.31 3.50 4.00 1.00 5.00 
6. Overall impression 3.31 1.08 3.50 4.00 2.00 5.00 


Social studies (n = 20) 


1. Effective writing task 3.35 1.63 3.00 5.00 1.00 5.00 
2. Standards alignment 3.55 0.83 3.00 3.00 2.00 5.00 
3. Fidelity to LDC instruction 3.65 0.75 4.00 4.00 2.00 5.00 
4. Quality instructional strategies 3.70 0.66 4.00 4.00 3.00 5.00 
5. Coherence/clarity of module 3.75 1.12 4.00 4.00 0.00 5.00 
6. Overall impression 3.75 0.55 4.00 4.00 3.00 5.00 


Overall (n = 64) 


1. Effective writing task 3.48 1.27 4.00 3.00 1.00 5.00 
2. Standards alignment 3.47 0.98 3.00 3.00 1.00 5.00 
3. Fidelity to LDC instruction 3.64 1.03 4.00 4.00 1.00 5.00 
4. Quality instructional strategies 3.75 1.02 4.00 4.00 1.00 5.00 
5. Coherence/clarity of module 3.75 1.14 4.00 4.00 0.00 5.00 
6. Overall impression 3.69 0.89 4.00 4.00 1.00 5.00 
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Table F11 


Distribution (Percentage) of Ratings for the Secondary Modules by Content Area 


Dimension 1 2 3 4 5 
ELA (n = 28) 
1. Effective writing task 7.14 10.71 28.57 28.57 25.00 
2. Standards alignment 3.57 3.57 42.86 35.71 14.29 
3. Fidelity to LDC instruction 10.71 3.57 17.86 42.86 25.00 
4. Quality instructional strategies 7.14 0.00 21.43 28.57 42.86 
5. Coherence/clarity of module 3.57 0.00 14.29 46.43 35.71 
6. Overall impression 3.57 3.57 17.86 53.57 21.43 
Science (n = 16) 
1. Effective writing task 0.00 12.50 31.25 43.75 12.50 
2. Standards alignment 0.00 43.75 6.25 31.25 18.75 
3. Fidelity to LDC instruction 6.25 0.00 43.75 31.25 18.75 
4. Quality instructional strategies 6.25 12.50 31.25 37.50 12.50 
5. Coherence/clarity of module 12.50 25.00 12.50 37.50 12.50 
6. Overall impression 0.00 31.25 18.75 37.50 12.50 
Social studies (n = 20) 
1. Effective writing task 25.00 0.00 30.00 5.00 40.00 
2. Standards alignment 0.00 5.00 50.00 30.00 15.00 
3. Fidelity to LDC instruction 0.00 5.00 35.00 50.00 10.00 
4. Quality instructional strategies 0.00 0.00 40.00 50.00 10.00 
5. Coherence/clarity of module 0.00 0.00 25.00 50.00 20.00 
6. Overall impression 0.00 0.00 30.00 65.00 5.00 
Overall (n = 64) 
1. Effective writing task 10.94 7.81 29.69 25.00 26.56 
2. Standards alignment 1.56 14.06 35.94 32.81 15.63 
3. Fidelity to LDC instruction 6.25 3.13 29.69 42.19 18.75 
4. Quality instructional strategies 4.69 3.13 29.69 37.50 25.00 
5. Coherence/clarity of module 4.69 6.25 17.19 45.31 25.00 
6. Overall impression 1.56 9.38 21.88 53.13 14.06 
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Table F12 
Descriptive Statistics for the Secondary Modules by Cohort 


Dimension M SD Median Mode Minimum Maximum 


Cohort 1 returning (n = 15) 


1. Effective writing task 3.67 1.23 4.00 3.00 1.00 5.00 
2. Standards alignment 3.80 0.94 4.00 3.00 2.00 5.00 
3. Fidelity to LDC instruction 3.40 1.06 4.00 4.00 1.00 4.00 
4. Quality instructional strategies 3.87 1.06 4.00 4.00 2.00 5.00 
5. Coherence/clarity of module 3.87 1.19 4.00 4.00 1.00 5.00 
6. Overall impression 3.87 0.83 4.00 4.00 2.00 5.00 


Cohort 1 new (n =7) 


1. Effective writing task 4.71 0.49 5.00 5.00 4.00 5.00 
2. Standards alignment 4.14 0.69 4.00 4.00 3.00 5.00 
3. Fidelity to LDC instruction 4.14 0.90 4.00 5.00 3.00 5.00 
4. Quality instructional strategies 4.14 0.69 4.00 4.00 3.00 5.00 
5. Coherence/clarity of module 4.00 1.83 5.00 5.00 0.00 5.00 
6. Overall impression 4.29 0.49 4.00 4.00 4.00 5.00 
Cohort 2 (n = 42) 

1. Effective writing task 3.21 1.26 3.00 3.00 1.00 5.00 
2. Standards alignment 3.24 0.96 3.00 3.00 1.00 5.00 
3. Fidelity to LDC instruction 3.64 1.03 4.00 4.00 1.00 5.00 
4. Quality instructional strategies 3.64 1.06 4.00 3.00 1.00 5.00 
5. Coherence/clarity of module 3.67 1.00 4.00 4.00 1.00 5.00 
6. Overall impression 3.52 0.92 4.00 4.00 1.00 5.00 


Overall (n = 64) 


1. Effective writing task 3.48 1.27 4.00 3.00 1.00 5.00 
2. Standards alignment 3.47 0.98 3.00 3.00 1.00 5.00 
3. Fidelity to LDC instruction 3.64 1.03 4.00 4.00 1.00 5.00 
4. Quality instructional strategies 3.75 1.02 4.00 4.00 1.00 5.00 
5. Coherence/clarity of module 3.75 1.14 4.00 4.00 0.00 5.00 
6. Overall impression 3.69 0.89 4.00 4.00 1.00 5.00 
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Table F13 


Distribution (Percentage) of Ratings for the Secondary Modules by Cohort 


Dimension 1 2 3 4 5 
Cohort 1 returning (n = 15) 
1. Effective writing task 6.67 6.67 33.33 20.00 33.33 
2. Standards alignment 0.00 6.67 33.33 33.33 26.67 
3. Fidelity to LDC instruction 13.33 0.00 20.00 66.67 0.00 
4. Quality instructional strategies 0.00 13.33 20.00 33.33 33.33 
5. Coherence/clarity of module 6.67 6.67 13.33 40.00 33.33 
6. Overall impression 0.00 6.67 20.00 53.33 20.00 
Cohort 1 new (n = 7) 
1. Effective writing task 0.00 0.00 0.00 28.57 71.43 
2. Standards alignment 0.00 0.00 14.29 57.14 28.57 
3. Fidelity to LDC instruction 0.00 0.00 28.57 28.57 42.86 
4. Quality instructional strategies 0.00 0.00 14.29 57.14 28.57 
5. Coherence/clarity of module 0.00 0.00 0.00 28.57 57.14 
6. Overall impression 0.00 0.00 0.00 85.71 28.57 
Cohort 2 (n = 42) 
1. Effective writing task 14.29 9.52 33.33 26.19 16.67 
2. Standards alignment 2.38 19.05 40.48 28.57 9.52 
3. Fidelity to LDC instruction 4.76 4.76 33.33 35.71 21.43 
4. Quality instructional strategies 7.14 0.00 35.71 35.71 21.43 
5. Coherence/clarity of module 4.76 7.14 21.43 50.00 16.67 
6. Overall impression 2.38 11.90 26.19 50.00 9.52 
Overall (n = 64) 
1. Effective writing task 10.94 7.81 29.69 25.00 26.56 
2. Standards alignment 1.56 14.06 35.94 32.81 15.63 
3. Fidelity to LDC instruction 6.25 3.13 29.69 42.19 18.75 
4. Quality instructional strategies 4.69 3.13 29.69 37.50 25.00 
5. Coherence/clarity of module 4.69 6.25 17.19 45.31 25.00 
6. Overall impression 1.56 9.38 21.88 53.13 14.06 


162 


Table F14 


Descriptive Statistics for the Exploratory Analysis of Elementary Modules 


Dimension M SD Median Mode Minimum Maximum 


2016-2017 (n = 7) 


1. Effective writing task 3.43 1.40 4.00 2.00 2.00 5.00 
2. Standards alignment 3.00 1.00 3.00 3.00 2.00 5.00 
3. Fidelity to LDC instruction 3.43 1.40 4.00 2.00 2.00 5.00 
4. Quality instructional strategies 3.43 1.40 4.00 2.00 2.00 5.00 
5. Coherence/clarity of module 3.86 0.90 4.00 3.00 3.00 5.00 
6. Overall impression 3.43 1.40 4.00 2.00 2.00 5.00 


2017-2018 (n = 7) 


1. Effective writing task 3.86 0.38 4.00 4.00 3.00 4.00 
2. Standards alignment 4.14 0.69 4.00 4.00 3.00 5.00 
3. Fidelity to LDC instruction 4.14 1.07 4.00 4.00 2.00 5.00 
4. Quality instructional strategies 3.14 1.46 4.00 4.00 1.00 5.00 
5. Coherence/clarity of module 3.71 1.11 4.00 3.00 2.00 5.00 
6. Overall impression 3.14 1.46 4.00 4.00 0.00 4.00 
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Table F15 


Distribution (Percentage) of Ratings for the Exploratory Analysis of Elementary Modules 


Dimension 1 2 3 4 5 

2016-2017 (n = 7) 

1. Effective writing task 0.00 42.86 0.00 28.57 28.57 
2. Standards alignment 0.00 28.57 57.14 0.00 14.29 
3. Fidelity to LDC instruction 0.00 42.86 0.00 28.57 28.57 
4. Quality instructional strategies 0.00 42.86 0.00 42.86 28.57 
5. Coherence/clarity of module 0.00 0.00 42.86 28.57 28.57 
6. Overall impression 0.00 42.86 0.00 28.57 28.57 
2017-2018 (n = 7) 

1. Effective writing task 0.00 0.00 14.29 85.71 0.00 
2. Standards alignment 0.00 0.00 14.29 57.14 28.57 
3. Fidelity to LDC instruction 0.00 14.29 0.00 42.86 42.86 
4. Quality instructional strategies 14.29 28.57 0.00 42.86 14.29 
5. Coherence/clarity of module 0.00 14.29 28.57 28.57 28.57 
6. Overall impression 0.00 0.00 28.57 57.14 0.00 
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Appendix G: 


Fidelity Matrix 


1. Key Component = Common Planning time for LDC Professional Learning Community with Synchronous Coach Support 


Teacher Expectation that 
Attendance at — PLC teachers will 
weekly PLC regularly attend 
meetings PLC meetings. 


Teacher 


PLC 
Reflection 
form will 
include 
attendance 
record for 
both coach- 
facilitated 
and teacher 
leader- 
facilitated 
PLC 
meetings 


Reflection form 
will be filled out 
by Teacher 
Leader for 
each weekly 
PLC session. 
LDC will 
deliver 
attendance 
data to 
CRESST twice 
per year. 


0 (very low) = less 
than 70% 
attendance at PLC 
sessions 

1 (low) = 70-79% 
attendance at PLC 
sessions 

2 (moderate) = 80- 
89% attendance at 
PLC sessions 

3 (ideal) = at least 
90% attendance at 
PLC sessions 


Adequate 
implementation at 
teacher level is score 
of 2 


School-level: 

0 (very low) = less 
than 65% of PLC 
teachers in school 
with score of 2 

1 (low) = 65% to 
<75% of PLC 
teachers in school 
with score of 2 

2 (moderate) = 75- 
85% of PLC teachers 
in school with score 
of 2 

3 (high) = over 85% 
of PLC teachers in 
school with score of 2 


Adequate 
implementation at 
school level is score 
of 2. 


Sample-level 

0= < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score = 2 

3 => 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 3 


All participating 
schools 


Once per year from 
2017-18 to 2018-19 
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Amount of time 
spent on LDC 
during common 
planning time 


PLCs expected to 
spend at least 60 
minutes per PLC 
meeting where the 
coach joins. 


School 


PLC 
reflection 
form 


Teacher 
Leader will 
note time spent 
on LDC in the 
same reflection 
form that 
captures 
attendance. 
Data delivered 
to CRESST 
twice per year. 


0 (low) = modal* 
response of under 
45 minutes per 
push-in session 

1 (moderate) = 
modal response of 
45-59 minutes per 
push-in session 

2 (high) = modal 
response of 60-74 
minutes per push-in 
session 

3 (ideal) = modal 
response of 75-90 
minutes per push-in 
session 


*if there is more 
than one modal 
response, the 
highest modal 
response will be 
used. 


Adequate 
implementation at 
school level is score 
of 2 


Sample level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score = 2 

3 => 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 3 


All participating 
schools 


Once per year from 
2017-18 to 2018-19 
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Exposure to 
LDC LEARN 
content during 
first 
instructional 
cycle 


Coaches are 


expected to use 
Instructional Cycles 
in their biweekly 
digital push-in 
sessions with the 
PLCs. 


Teacher 


CoreTools 
Data 


Users view 
Instructional 
Cycle sessions 
during coach 
push-in 
sessions. Data 
delivered to 
CRESST twice 
per year. 


0 (very low) = 
teacher views less 
than 50% of 
sessions in the 
Instructional Cycle 
1 (low) = teacher 
views between 
50% and less than 
60% of sessions in 
the Instructional 
Cycle 

2 (moderate) 
teacher views 
between 60% and 
less than 70% of 
sessions in the 
Instructional Cycle 
3 (high) teacher 
views 70% or more 
of sessions in the 
Instructional Cycle 


Adequate 
implementation at 
school level is score 
of 2. 


School-level: 0 (very 
low) = less than 65% 
of PLC teachers in 
school with score of 2 
1 (low) = 65% to 
<75% of PLC 
teachers in school 
with score of 2 2 
(moderate) = 75-85% 
of PLC teachers in 
school with score of 2 
3 (high) = over 85% 
of PLC teachers in 
school with score of 2 
Adequate 
implementation at 
school level is score 
of 2. 


Sample level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score = 2 

3 => 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 3 


All participating 
schools 


Once per year from 
2017-18 to 2018-19 
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Exposure to 
LDC LEARN 
content during 
second 
instructional 
cycle 


Coaches are 


expected to use 
Instructional Cycles 
in their biweekly 
digital push-in 
sessions with the 
PLCs. 


Teacher 


CoreTools 
Data 


Users view 
Instructional 
Cycle sessions 
during coach 
push-in 
sessions. Data 
delivered to 
CRESST twice 
per year. 


0 (very low) = 
teacher views less 
than 50% of 
sessions in the 
Instructional Cycle 
1 (low) = teacher 
views between 
50% and less than 
60% of sessions in 
the Instructional 
Cycle 

2 (moderate) 
teacher views 
between 60% and 
less than 70% of 
sessions in the 
Instructional Cycle 
3 (high) teacher 
views 70% or more 
of sessions in the 
Instructional Cycle 


Adequate 
implementation at 
school level is score 
of 2. 


School-level: 0 (very 
low) = less than 65% 
of PLC teachers in 
school with score of 2 
1 (low) = 65% to 
<75% of PLC 
teachers in school 
with score of 2 2 
(moderate) = 75-85% 
of PLC teachers in 
school with score of 2 
3 (high) = over 85% 
of PLC teachers in 
school with score of 2 
Adequate 
implementation at 
school level is score 
of 2. 


Sample level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score = 2 

3 => 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 3 


All participating 
schools 


Once per year from 
2017-18 to 2018-19 
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Perceived 
effectiveness of 
engagement in 
PLC on teacher 
competencies 


Common planning 
time is expected to 
lead to teacher 
proficiency in 4 key 
competencies: 
identifying 
standards-aligned 
assignments, 
construction of 
quality assignment 
prompts, 
developing 
instructional plans, 
and formative 
assessment. 


Teacher 


Teacher 
survey 


Teacher 
Survey asks 
respondents to 
report 
improvement 
from the 
beginning to 
the end of the 
year. There are 
7 items 
covering the 4 
competencies. 
Calculations 
for this 
measure are 
based on an 
index 
averaging the 
seven items. 


0 (very low) = 
teacher reports that 
on average her 
abilities did not 
improve at all 
between the 
beginning and end 
of the year’s work 
with LDC 

1 (low) = teacher 
reports that on her 
abilities improved a 
little 

2 (moderate) = 
teacher reports that 
her abilities 
improved 
moderately 

3 (high) = teacher 
reports her abilities 
improved a great 
deal 


Adequate 
implementation is 
score of 2 at teacher 
level 
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School-level: 

0 (very low) = less 
than 65% of PLC 
teachers in school 
with score of 2 

1 (low) = 65% to 
<75% of PLC 
teachers in school 
with score of 2 

2 (moderate) = 75- 
85% of PLC teachers 
in school with score 
of 2 

3 (high) = over 85% 
of PLC teachers in 
school with score of 2 


Adequate 
implementation at 
school level is score 
of 2. 


Sample level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score = 2 

3 => 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 3 


All participating 
schools 


Once per year from 
2016-17 to 2018-19 


A ate 


Coach 
Comments on 
Modules 


Coaches required 
to provide feedback 
on modules at set 
points in the 
instructional 
sequence: teaching 
task, complete 
module, and 
revised module. 


Module 
(universe is 
all modules 
linked to 
LEARN tab 
courses) 


CoreTools 
Analytic 
data 


LDC.org will 
provide at least 
2 data pulls per 
year of 
individual 
teacher level 
CoreTools data 
with 
information on 
teachers’ 
interaction with 
the online 
courses and 
CoreTools 
features 


0 (very low) = no 
coach comments 

1 (low) = coach 
comment provided 
at one key point in 
design 

2 (moderate) = 
coach comment 
provided at two key 
points in design 

3 (high) = coach 
comment provided 
at three or more 
key points in design 


Adequate 
implementation at 
module level is score 
of 2 


School-level: 0 (very 
low) = less than 65% 
of modules from 
school PLC with 
score of 2 

1 (low) = 65-74% of 
modules from school 
PLC with score of 2 
2 (moderate) = 75- 
85% of modules from 
school PLC with 
score of 2 

3 (high) = over 85% 
of modules from 
school PLC with 
score of 2 Adequate 
implementation at 
school level is score 
of 2. 


Sample-level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score = 2 

3 => 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 2 


All participating 
schools 


Once per year from 
2017-18 to 2018-19 
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Coach 
Formative Peer 
Review on 
Modules 


Coaches required 
to provide feedback 
on modules via 
Peer Review 
Feedback at set 
points in the 
instructional 
sequence: teaching 
task, complete 
module, and 
revised module. 


Teacher 
(universe is 
all modules 
linked to 
LEARN tab 
courses) 


CoreTools 
Analytic 
data 


LDC.org will 
provide at least 
2 data pulls per 
year of 
individual 
teacher level 
CoreTools data 
with 
information on 
teachers’ 
interaction with 
the online 
courses and 
CoreTools 
features 


0 (low) = coach 
feedback not 
provided on a 
linked course via 
peer review rubric 
at least once per 
year per teacher 

1 (moderate) = 
coach feedback 
provided on a 
linked course once 
via peer review 
rubric per year per 
teacher 


Adequate 
implementation at 
teacher level is score 
of 1 


School-level: 

0 (very low) = less 
than 65% of teachers 
from school PLC with 
score of 1 

1 (low) = 65-74% of 
teachers from school 
PLC with score of 1 
2 (moderate) = 75- 
85% of teachers from 
school PLC with 
score of 1 

3 (high) = over 85% 
of teachers from 
school PLC with 
score of 1 Adequate 
implementation at 
school level is score 
of 2. 


Sample-level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score = 2 

3 => 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 2 


All participating 
schools 


Once per year from 
2017-18 to 2018-19 
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Teacher 
perception of 
the helpfulness 
of coach written 
feedback on 
modules 


Teacher rating of 
coach-provided 
asynchronous 
feedback via 
CoreTools between 
push-in sessions 


Teacher 


Teacher 
survey 


Spring teacher 
survey 
administered 
by CRESST 
will ask 
teachers to 
report on 
whether they 
found written 
feedback via 
the peer review 
form and 
comments in 
CoreTools 
helpful. 


0 (very low) = not 
helpful 

1 (low) = alittle 
helpful 

2 (moderate) = 
moderately helpful 
3 (high) = very 
helpful 


Adequate 
implementation at 
teacher level is score 
of 2 
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School-level: 

0 (very low) = less 
than 65% of teachers 
in school PLC with 
score of 2 

1 (low) = 65-74% of 
teachers in school 
PLC with score of 2 
2 (moderate) = 75- 
85% of teachers in 
school PLC with 
score of 2 

3 (high) = over 85% 
of teachers in school 
PLC with score of 2 


Adequate 
implementation at 
school level is score 
of 2. 


Sample-level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score = 2 

3 => 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 2 


All participating 
schools 


Once per year from 
2016-17 to 2018-19 


f\ 


Module editing 


Participating 
teachers expected 
to edit at least one 
module in each 
year of 
implementation. 


Teacher 


CoreTools 
data 


Data 
transmitted by 
LDC.org to 
CRESST in 2 
data pulls per 
year 


0 = no evidence of 
edits to task on any 
module 

1 = edited task on 
at least one module 
2 = edited task in at 
least one module 
and either 
standards OR text 
in at least one 
module 

3 = edited task in at 
least one module 
and either 
standards OR text 
in at least one 
module, and either 
skills and mini- 
tasks or rubric in at 
least one module. 


Adequate 
implementation at 
teacher level is score 
of 2 


School-level: 

0 (very low) = less 
than 65% of teachers 
in school PLC with 
score of 2 

1 (low) = 65% to 
<75% of teachers in 
school PLC with 
score of 2 

2 (moderate) = 75- 
85% of teachers in 
school PLC with 
score of 2 

3 (high) = over 85% 
of teachers in school 
PLC with score of 2 


Adequate 
implementation at 
school level is score 
of 2. 


Sample-level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score = 2 

3 => 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 3 


All participating 
schools 


Once per year from 
2017-18 to 2018-19 
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Module 
implementation 


Participating 
teachers expected 
to implement two 
modules per year 
and upload student 
work based on 
modules 


Teacher 


CoreTools 
data 


Data 
transmitted by 
LDC.org to 
CRESST in 2 
data pulls per 
year 


0 (very low) = no 
student work 
samples uploaded 
to CoreTools 

1 (low) = student 
work samples for 1 
module uploaded 
2 (moderate) = 
student work 
samples for 2 
modules uploaded 
3 (high) = student 
work samples for 3 
or more modules 
uploaded 


Adequate 
implementation at 
teacher level is score 
of 2 


0 (very low) = less 
than 65% of teachers 
in school PLC with 
score of 2 

1 (low) = 65-74% of 
teachers in school 
PLC with score of 2 
2 (moderate) = 75- 
85% of teachers in 
school PLC with 
score of 2 

3 (high) = over 85% 
of teachers in school 
PLC with score of 2 


Adequate 
implementation at 
school level is score 
of 2. 


Sample-level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score = 2 

3 => 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 3 


All participating 
schools 


Once per year from 
2017-18 to 2018-19 
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- e O ole 


Frequency of =| At minimum, School PLC 0 (very low) = less | Adequate Sample-level All participating | Once per year from 
CoachiTeacher | Teacher Leaders Reflection than 4 planning and implementation at 0 =< 25% schools with | Schools 2017-18 to 2018-19 
Leader monthly | are expected to form progress callsin —_| school level is score score > 2 
meetings have 30 minute the year of 2 1 = 26-50% schools with 
planning ee re 1 (low) = 4-8 score > 2 
progress call witl planning and - A 
2 = 51-75% schools with 
coach each month. progress calls score = 2 
Ct a hai 3 = > 75% schools with 
13 planning and score > 2 
progress calls 
3 (high) = 14 or : ; 
more planning and Sia nicer 
progress calls : “ead e level is a score 
Administrator |One administrator | School LDC 0 (very low) = Adequate Sample-level All participating | Once per year from 
attendance at | per school is Attendance participating in no implementation at 0 =< 25% schools with Schools 2017-18 to 2018-19 
quarterly in- expected to attend Records tie school level is score score > 2 
person LDC’s in-person ow)= of 2 1 = 26-50% schools with 
administrator | administrator participating in 1 score > 2 ° 
meeting meetings, occurring event - 


three times during 
the school year. 


2 (moderate) = 
participating in 2 
events 

3 (high) = 
participating in 3-4 
events 


2 = 51-75% schools with 
score = 2 

3 => 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 3 
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Teacher Leader 
attendance at 
quarterly in- 
person Teacher 
Leader 
meetings 


Teacher Leader is 
expected to attend 
LDC’s in-person 
administrator 
meetings, occurring 
three times during 
the school year. 


School 


LDC 
Attendance 
records 


0 (very low) = 
participating in no 
event 

1 (low) = 
participating in 1 
event 

2 (moderate) = 
participating in 2 
events 

3 (high) = 
participating in 3-4 
events 


Adequate 
implementation at 
school level is score 
of 2 


Sample-level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score = 2 

3 => 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 3 


All participating 
schools 


Once per year from 
2017-18 to 2018-19 
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Principal Mini- 
task 
Observation 


School leaders 
expected to 
observe all LDC 
teachers 
implementing at 
least one mini-task 


Teacher 


Teacher 
survey 


Data collected 
each Spring 
via CRESST 
survey. 


0 (low) = teacher 
reports 0 
observations by 
school leader 

1 (moderate) = 
teacher reports 1 
observation by 
school leader 

2 (high) = teacher 
reports 2 
observations by 
school leader 

3 (very high) = 
teacher reports 3 or 
more observations 
by school leader 


Adequate 
implementation at 
teacher level is 1 


0 (very low) = less 
than 65% of teachers 
in schoo] PLC 
reporting adequate 
implementation 

1 (low) = 65% to 74% 
of teachers in school 
PLC reporting 
adequate 
implementation 

2 (moderate) = 75% 
to 99% of teachers in 
school PLC reporting 
adequate 
implementation 

3 (high) = 100% of 
teachers in school 
PLC reporting 
adequate 
implementation 


Adequate 
implementation at 
school level is score 
of 2. 


Sample-level 

0 = < 25% schools with 
score = 2 

1 = 26-50% schools with 
score = 2 

2 = 51-75% schools with 
score = 2 

3 => 75% schools with 
score = 2 


Adequate implementation 
at sample level is a score 
of 3 


All participating 
schools 


Once per year from 
2016-17 to 2018-19 
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Appendix H: 
Outcome Analysis Methodology 


Analysis Model Specification 


For our outcome analyses, we used a threshold of p < .05 to determine whether there was 
a Statistically significant impact of LDC on ELA achievement. In addition to the LDC treatment 
indicator, a teacher effect for years of experience was included, as well as an aggregate 
indicator measuring the mean baseline performance of each student’s classroom peers. The 
fixed effects also included student characteristics to identify the matched comparison sample of 
students, such as baseline achievement, socioeconomic status, demographics, language 
proficiency, grade level, and participation in special education. 


The three-level MMMC model was used to estimate the impacts of the LDC intervention 
on student learning. This same analytic model will be used to estimate impacts for future years 
of LDC. The general specification for the middle school level MMMC model is shown in the 
following equation using similar notation proposed by Browne et al. (2001, equation 6) and 
applied in Tranmer, Steel, and Browne (2014, equation 3). 


_ our (3) (2) 
yi = xB + Uschool(i) > Wi jUj te; 


j€Teacher(i) 


i = 1,..,n Teacher(i) Cc (1,...,J) 
(3) (2) 
Uschool(i) ~ N(0, 022), uj ~ N(0, 0202); e; ~ N(0,02) 


In this model y; is the student achievement score response, Xiis a vector of the fixed 
covariates and f is the vector of the corresponding fixed effects. School(i) is the school which 
student i attends, thus the term iG represents the random effects for that level of 
classification. Within the term > jereacner(i) wi ju”, uy? is the set of j random effects for the 
teachers included in the selected dataset, and w; ; is the weight which sums to 1 for each 
student applied in proportion to the instruction time assigned with each teacher. The following 


presents an example of the full model middle school specification. 


achievement; = By + LDC; * 6, + priorELA; * B2 + priorMath, * B3 + Female; * 6, + LEP; 
* Bs + SES; * Bg + Black; * B; + Hispanic; * Bg * +SPED; * Bo + Gifted; * By 
+ HonorsELA; * Bi, + priorClassELA; * Bz + TeachExp; * By3 


+ TeachAttend; * By, + aan > wi jy” +e; 


j€Teacher(i) 


(3) (2) 
Uschool(i) ~ N(0, o%¢3)), ue N(0, 0202) e; ~ N(0,02) 
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In this model, achievement; is the standardized ELA outcome score for student i; In the 
dosage-dependent model LDC; is the proportion of core class instruction time taken with an 
LDC teacher (ranges 0-1). In the dosage-independent model LDC; is coded as zero for 
comparison students and as 1 for students receiving any level of LDC teacher exposure. 


Female;, LEP;, Black;, Hispanic; SPED;, Gifted;,and SES; are student demographic 
indicators coded 1 if the status is present and 0 if absent; 


HonorsELA; is dummy coded 1 when a student was enrolled in an at least one 
honors ELA course and otherwise as 0, allowing students who did not enroll in any 
honors ELA courses to serve as the reference group; 


priorELA, and priorMath, are standardized student achievement scores from the 
baseline year; 


priorClassELA; is the aggregated mean of the baseline ELA scores for all the core 
class peers of student i; 


TeachExp; is the aggregated percentage of the years of teaching experience less than 
three for those teachers that student i was exposed to in his/her core classes; 


TeachAttend; is the aggregated mean attendance for those teachers that student i 
was exposed to in his/her core classes; 


B, is the impact of LDC, the treatment; 

B12 is the average difference between Cohort 1 and Cohort 2; 
Bo & B3 are the effects of the baseline score covariates; 

By ... By are the effects of the demographic covariates; 


B41 --- P14 are the effects of the aggregated class level covariates; 


Pr 
School(i)’ 
respectively assumed to all have a mean of zero and a variance, ato One: ee 


respectively. 


i”, e; are the error components at the school, teacher, and student level 


Student/Teacher Course Exposure Weighting 


Tables H1 and H2 demonstrate how the process of calculating general MMMC teacher 
weights and LDC treatment weights, for the dosage modeling approach, was conducted 
respectively for elementary and middle school. In elementary school, in the event that a 
student was exposed to more than one teacher, each content area was given equal weight in 
distributing teacher/student exposure. For example, if a student was enrolled for both ELA and 
social studies/history under one teacher, then that teacher was coded as .67 for having 
contributed to two thirds of the students’ core curriculum exposure. If the same student 
enrolled in science with a different teacher than the one who was linked to their course marks 
in ELA and social studies/history, then that science teacher would have been coded as .33 and 
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all other teachers in the sample would have been coded as zero. This would then result in the 
student’s exposure adding to a unity (1). 


Table H1 
Example of Elementary School Student/Teacher Weighting Based on Course Links 


Marking Example student/teacher weighting for use in MMMC Example treatment dosage 
period (Weight = Marking Period/Total Marking Period3) weight 
One Student enrolled with an intervention teacher: Weight = .333 


Weight = (1/3) = .333 

Two Student enrolled with an intervention Teacher: Weight = .333 
Weight = (1/3) = .333 

Three Student enrolled with a non-intervention teacher: Weight = .0 
Weight = (1/3) = .333 


Total Unity: for every student the student/teacher weights sum Treatment weight = .667 
to1 


Note. In the selected samples for these analyses, core content courses within each marking period were taught by 
a single teacher. 


In middle school, students’ exposure to teachers at the course level in the three core 
content areas was coded in the same manner as in the elementary grades based on enrolled 
time preceding the assessment period. A difference in our middle school coding process was 
that we did not force each core content area into equal weighting. Instead, each core content 
area exposure contributed to a core content area total sum that formed the basis from which 
the weights were proportioned. 


Most commonly a student had equivalent days of core instruction exposure in each of the 
three content areas (often 214 days in each content area). In that scenario, if a student had 
exposure to three different teachers, then each teacher would contribute one third (.33) of the 
overall core curriculum exposure and all other teachers in the sample would be coded as zero. 
However, in addition to the typical core science course, extra core science courses were also 
included in the LDC analysis (e.g., an eighth-grade student taking biology), which made it 
possible then for a student to accumulate more units in science than in the other two content 
areas. 


The weighting in middle school was always distributed as a proportion of the total 
exposure days in the three content areas. Therefore, if a student accumulated 300 science days 
(across two courses), 200 social studies days, and 200 ELA days, the base number of instruction 
days would be 700 days. If, using that same scenario, the same teacher taught both the typical 
core and biology courses then that teacher would contribute three sevenths (.43) of the overall 
core curriculum exposure with the social studies and science teachers each contributing the 
other two sevenths (.285), again resulting in the student’s exposure adding to a unity (1). 
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Table H2 
Example of Middle School Student/Teacher Weighting Based on Course Mark Links 


Core content Example student/teacher weighting for use in Example treatment 
area Course name MMMC (Weight = Subject Days/Total Days) dosage weight 
ELA English 7A& 7B Student enrolled two terms of core ELA (Grade Weight = .400 


7) with an intervention teacher: 
Weight = 2/5 = .400 


Social studies Social Studies Student enrolled two terms of core social Weight = .0 
WHG: ANC CIV & studies (Grade 7) with a nonintervention 
B teacher: 


Weight = 2/5 = .400 


Science Science 7 Student enrolled one terms of core science Weight = .0 
(Grade 7) with a nonintervention teacher: 
Weight = 1/5 = .200 

Total Unity: for every student the student/teacher Treatment weight 
weights sum to 1 = 400 


Note. “Days” refers to core content enrolled days preceding the assessment date. 


Calculation of Effect Size 


We calculated student-level effect sizes according to the What Works Clearinghouse 
(WWC) 3.0 criteria. Specifically, for the impact analysis with treatment status as a dichotomous 
variable, we calculated Hedges’ g, the difference in adjusted mean outcomes for the groups 
divided by the unadjusted pooled within-group standard deviation of the outcome measure in 
the sample, for all outcomes. The difference in adjusted mean outcomes is estimated by f, in 
the models we previously defined, as outlined in the WWC standards handbook for computing 
effect sizes in multi-level frameworks. Specifically, 


@y 


I= 
(ni — 1)s? + (nm, — 1)s? 
nitn, — 2 


where y is f£,, which is the coefficient from the MMMC for the intervention effect. 


Note that in our analyses the outcome measure is standardized within the analytical 
sample (M = 0, SD = 1). As a result, we expect that g would likely be quite similar to the f, 
coefficient from the MMMC model in the large samples we plan to collect later in the study. 


Though it is not standard to use g with a continuous treatment effect, as in the case of 
our dosage-dependent treatment measure, we have defined and matched populations n; and 
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n, where treated students (n;) could have any positive treatment value <= 1, and comparison 
students (n,) had a treatment value of zero. We could therefore calculate g in the case of our 
dosage-dependent treatment measure, and again expect that it would not differ substantially 
from the f, coefficient. It is crucial, however, to note that g and (3, in the dosage-dependent 
models reflect the effect size projected for a student who would receive exposure to 
intervention teachers in all of their core classes. Along with this effect we report average 
dosage received by treated students so that the average treatment effect on treated students 
could be calculated. 
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